Mercurial > projects > dang

annotate lexer/Lexer.d @ 88:eb5b2c719a39 new_gen

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Major change to locations, tokens and expressions. A location (now SourceLocation or SLoc) is only 32 bit in size - disadvantage is that it can't find its own text. You have to go through the new SourceManager to do that. This has caused changes to a lot of stuff and removal of DataSource and the old Location Additionally Exp has gotten some location stuff, so we can give proper error messages. Not in Decl and Stmt yet, but thats coming too.
author Anders Halager <halager@gmail.com>
date Sun, 04 May 2008 18:13:46 +0200
parents 192da4976daa
children a49bb982a7b0
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
1 module lexer.Lexer;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
2
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
3 import misc.Error,
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
4 basic.SourceManager;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
5
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
6 import lexer.Token,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
7 lexer.Keyword;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
8
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
9 import tango.io.Stdout;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
10
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
11 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
12 The Lexer class will supply you with methods to tokenize a D file. Supply the
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
13 Lexer with a DataSource and you can 'peek' and 'next' Tokens from the file.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
14
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
15 For more info about Tokens, look up the lexer.Token module.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
16 */
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
17 class Lexer
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
18 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
19 public:
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
20
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
21 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
22 Create a new Lexer.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
23 */
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
24 this(SourceLocation start, SourceManager src_mgr)
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
25 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
26 sm = src_mgr;
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
27 start_loc = start;
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
28 position = 0;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
29 source = sm.getRawData(start_loc);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
30
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
31
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
32 charTable.length = 256;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
33 foreach (c; "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_")
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
34 charTable[c] = CharType.Letter;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
35
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
36 foreach (c; "0123456789")
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
37 charTable[c] = CharType.Number;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
38
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
39 foreach (c; "(){}[];:.,=!<>+-*/%")
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
40 charTable[c] = CharType.Symbol;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
41
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
42 foreach (c; " \n")
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
43 charTable[c] = CharType.Whitespace;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
44
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
45 symbolFunctions.length = 256;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
46
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
47 symbolFunctions['('] = &openParentheses;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
48 symbolFunctions[')'] = &closeParentheses;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
49 symbolFunctions['{'] = &openBrace;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
50 symbolFunctions['}'] = &closeBrace;
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
51 symbolFunctions['['] = &openBracket;
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
52 symbolFunctions[']'] = &closeBracket;
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
53 symbolFunctions[';'] = &seperator;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
54 symbolFunctions[':'] = &colon;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
55 symbolFunctions['.'] = &dot;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
56 symbolFunctions[','] = &comma;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
57 symbolFunctions['='] = &eq;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
58 symbolFunctions['!'] = &ne;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
59 symbolFunctions['<'] = &le;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
60 symbolFunctions['>'] = &ge;
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
61 symbolFunctions['+'] = &plus;
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
62 symbolFunctions['-'] = &minus;
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
63 symbolFunctions['*'] = &star;
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
64 symbolFunctions['/'] = &slash;
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
65 symbolFunctions['%'] = &percent;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
66 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
67
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
68 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
69 Get the next token from the source. This method will move the
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
70 internal position forward to the next Token.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
71
43
a712c530b7cc A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents: 42
diff changeset
72 return: A Token - Token.type is TokType.EOF if there is
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
73 no more tokens in the file.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
74 */
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
75 Token next()
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
76 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
77 switch (getNextChar)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
78 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
79 case CharType.EOF:
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
80 SLoc loc;
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
81 return Token(Tok.EOF, loc, 0);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
82
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
83 case CharType.Whitespace:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
84 position += 1;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
85 return this.next;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
86
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
87 case CharType.Symbol:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
88 return lexSymbol;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
89
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
90 case CharType.Letter:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
91 return lexLetter;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
92
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
93 case CharType.Number:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
94 return lexNumber;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
95 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
96 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
97
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
98 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
99 Get the next token from the source. This method will NOT move the
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
100 internal position forward, and thereby having no side-effects.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
101
43
a712c530b7cc A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents: 42
diff changeset
102 return: A Token - Token.type is TokType.EOF if there is
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
103 no more tokens in the file.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
104 */
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
105 Token peek(int skip = 0)
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
106 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
107 int oldPosition = this.position;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
108 while (skip-- > 0)
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
109 this.next;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
110 Token t = this.next;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
111 this.position = oldPosition;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
112 return t;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
113 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
114
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
115 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
116 Return all errors that occurred while tokenizing the string.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
117
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
118 TODO: Error system not implemented yet - this is a stub!
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
119 */
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
120 public Error[] getErrors()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
121 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
122 return this.errors;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
123 }
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
124
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
125 private:
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
126 Token eq()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
127 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
128 if(source[position] == '=')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
129 return Token(Tok.Eq, Loc(position++ - 1), 2);
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
130 return Token(Tok.Assign, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
131 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
132 Token openBrace()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
133 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
134 return Token(Tok.OpenBrace, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
135 }
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
136 Token closeBrace()
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
137 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
138 return Token(Tok.CloseBrace, Loc(position - 1), 1);
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
139 }
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
140 Token openParentheses()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
141 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
142 return Token(Tok.OpenParentheses, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
143 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
144 Token closeParentheses()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
145 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
146 return Token(Tok.CloseParentheses, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
147 }
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
148 Token openBracket()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
149 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
150 return Token(Tok.OpenBracket, Loc(position - 1), 1);
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
151 }
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
152 Token closeBracket()
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
153 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
154 return Token(Tok.CloseBracket, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
155 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
156 Token seperator()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
157 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
158 return Token(Tok.Seperator, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
159 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
160 Token colon()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
161 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
162 return Token(Tok.Colon, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
163 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
164 Token dot()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
165 {
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
166 int pos = 0;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
167 while(getNextChar(0) == CharType.Number ||
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
168 this.source[position + pos + 1] == '_')
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
169 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
170 if(getNextChar(0) == CharType.Number)
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
171 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
172 position--;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
173 return lexNumber();
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
174 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
175 pos++;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
176 }
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
177 return Token(Tok.Dot, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
178 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
179 Token comma()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
180 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
181 return Token(Tok.Comma, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
182 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
183 Token ne()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
184 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
185 if(source[position] == '=')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
186 return Token(Tok.Ne, Loc(position++ - 1), 2);
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
187 return Token(Tok.Not, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
188 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
189 Token le()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
190 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
191 if(source[position] == '=')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
192 return Token(Tok.Le, Loc(position++ - 1), 2);
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
193 return Token(Tok.Lt, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
194 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
195 Token ge()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
196 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
197 if(source[position] == '=')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
198 return Token(Tok.Ge, Loc(position++ - 1), 2);
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
199 return Token(Tok.Gt, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
200 }
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
201 Token plus()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
202 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
203 return Token(Tok.Plus, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
204 }
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
205 Token minus()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
206 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
207 return Token(Tok.Minus, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
208 }
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
209 Token star()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
210 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
211 return Token(Tok.Star, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
212 }
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
213 Token slash()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
214 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
215 switch(source[position])
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
216 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
217 case '/':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
218 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
219 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
220 if(source[position++] == '\n')
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
221 return this.next;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
222 }
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
223 return Token(Tok.EOF, Loc(position), 0);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
224
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
225 case '*':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
226 position += 2;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
227 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
228 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
229 ++position;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
230 if(source[position-2] == '*')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
231 if(source[position-1] == '/')
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
232 return this.next;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
233 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
234 throw error(__LINE__, "Unexpected end of file. Unclosed comment block");
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
235
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
236 case '+':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
237 position += 2;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
238 int nesting = 1;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
239 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
240 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
241 ++position;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
242 if(source[position-2] == '+')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
243 if(source[position-1] == '/')
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
244 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
245 position++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
246 nesting--;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
247 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
248
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
249 if(source[position-2] == '/')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
250 if(source[position-1] == '+')
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
251 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
252 nesting++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
253 position++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
254 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
255
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
256 if(nesting == 0)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
257 return this.next;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
258 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
259 throw error(__LINE__, "Unexpected end of file. Unclosed comment block");
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
260
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
261 default:
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
262 return Token(Tok.Slash, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
263 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
264 }
62
78a6808b2e0f Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents: 43
diff changeset
265
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
266 Token percent()
62
78a6808b2e0f Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents: 43
diff changeset
267 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
268 return Token(Tok.Percent, Loc(position - 1), 1);
62
78a6808b2e0f Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents: 43
diff changeset
269 }
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
270
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
271 Token lexNumber ()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
272 {
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
273 bool sign = false;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
274 bool dot = false;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
275 bool e = false;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
276
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
277 int i = 0;
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
278
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
279 bool end = false;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
280 while(!end)
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
281 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
282 switch(getNextChar(i))
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
283 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
284 case CharType.Number:
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
285 break;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
286 case CharType.Symbol:
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
287 if(this.source[position+i] == '.')
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
288 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
289 if(dot)
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
290 throw error(__LINE__,"Only one '.' is allowed in an floating number")
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
291 .tok(Token(Tok.Float, Loc(position + i), 1));
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
292 dot = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
293 break;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
294 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
295 end = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
296 continue;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
297 case CharType.Letter:
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
298 if(this.source[position+i] == '_')
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
299 break;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
300 if (this.source[position+i] == 'e' ||
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
301 this.source[position+i] == 'E')
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
302 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
303 if (e)
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
304 throw error(__LINE__,"Only one '"~this.source[position+i]
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
305 ~"' is allowed in an floating number");
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
306 e = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
307 break;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
308 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
309 end = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
310 continue;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
311
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
312 default:
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
313 end = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
314 continue;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
315 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
316 i++;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
317 }
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
318
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
319 position += i;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
320
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
321 return Token(Tok.Integer, Loc(position - i), i);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
322 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
323
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
324 Token lexSymbol ()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
325 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
326 Token t = symbolFunctions[source[position++]]();
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
327
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
328 return t;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
329 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
330
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
331 Token lexLetter ()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
332 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
333 int i = 0;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
334 bool hasNumber = false;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
335 while (getNextChar(++i) == CharType.Letter ||
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
336 getNextChar(i) == CharType.Number)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
337 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
338 if (getNextChar(i) == CharType.Number)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
339 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
340 hasNumber = true;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
341 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
342 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
343
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
344 Token t = Token(Tok.Identifier, Loc(), i);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
345
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
346 if (!hasNumber)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
347 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
348 char[] str = source[position .. position + i];
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
349 if(str in keywords)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
350 t.type = keywords[str];
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
351 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
352
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
353 position += i;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
354
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
355 return t;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
356 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
357
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
358 CharType getNextChar(int offset = 0)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
359 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
360 if (position + offset >= this.source.length)
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
361 return CharType.EOF;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
362
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
363 char current = source[position + offset];
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
364
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
365 CharType c = charTable[current];
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
366
43
a712c530b7cc A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents: 42
diff changeset
367 if(c == CharType.INVALID)
a712c530b7cc A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents: 42
diff changeset
368 throw error(__LINE__, "Read invalid symbol: '%0'").arg(current);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
369
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
370 return c;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
371
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
372 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
373
29
41d23f2762c3 Merge, and updated Error class
Anders Halager <halager@gmail.com>
parents: 28
diff changeset
374 Error error(uint line, char[] msg)
41d23f2762c3 Merge, and updated Error class
Anders Halager <halager@gmail.com>
parents: 28
diff changeset
375 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
376 return (new Error(msg));//.loc(Loc(position));
29
41d23f2762c3 Merge, and updated Error class
Anders Halager <halager@gmail.com>
parents: 28
diff changeset
377 }
41d23f2762c3 Merge, and updated Error class
Anders Halager <halager@gmail.com>
parents: 28
diff changeset
378
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
379 private final SourceLocation Loc(int pos = -1)
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
380 {
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
381 if (pos < 0)
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
382 return start_loc + position;
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
383 return start_loc + pos;
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
384 }
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
385
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
386 SourceManager sm;
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
387 SourceLocation start_loc;
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
388 int position;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
389 char[] source;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
390 Error[] errors;
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
391 CharType[] charTable;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
392 Token delegate()[] symbolFunctions;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
393 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
394
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
395 enum CharType : ubyte
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
396 {
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
397 INVALID,
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
398 Letter,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
399 Number,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
400 Symbol,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
401 Whitespace,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
402
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
403 EOF
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
404 }
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
405