Mercurial > projects > dang

annotate lexer/Lexer.d @ 41:f977aa28eb32 new_gen

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Now using arrays insted of switch - should speed things up a notch!
author Anders Johnsen <skabet@gmail.com>
date Tue, 22 Apr 2008 19:30:51 +0200
parents ce17bea8e9bd
children 4e879f82dd64
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
1 module lexer.Lexer;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
2
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
3 import misc.Error,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
4 misc.DataSource;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
5
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
6 import lexer.Token,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
7 lexer.Keyword;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
8
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
9 import tango.io.Stdout;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
10
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
11 class Lexer
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
12 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
13 public:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
14 this (DataSource source)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
15 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
16 this.source = source;
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
17 position = 0;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
18
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
19
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
20 charTable.length = 256;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
21 foreach( char c ; "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
22 charTable[c] = CharType.Letter;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
23
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
24 foreach( char c ; "0123456789")
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
25 charTable[c] = CharType.Number;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
26
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
27 foreach( char c ; "(){};:.,=!<>+-*/")
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
28 charTable[c] = CharType.Symbol;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
29
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
30 foreach( char c ; " \n")
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
31 charTable[c] = CharType.Whitespace;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
32
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
33 symbolFunctions.length = 256;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
34
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
35 symbolFunctions['('] = &openParentheses;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
36 symbolFunctions[')'] = &closeParentheses;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
37 symbolFunctions['{'] = &openBrace;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
38 symbolFunctions['}'] = &closeBrace;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
39 symbolFunctions[';'] = &seperator;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
40 symbolFunctions[':'] = &colon;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
41 symbolFunctions['.'] = &dot;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
42 symbolFunctions[','] = &comma;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
43 symbolFunctions['='] = &eq;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
44 symbolFunctions['!'] = &ne;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
45 symbolFunctions['<'] = &le;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
46 symbolFunctions['>'] = &ge;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
47 symbolFunctions['+'] = &add;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
48 symbolFunctions['-'] = &sub;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
49 symbolFunctions['*'] = &mul;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
50 symbolFunctions['/'] = &div;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
51 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
52
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
53 Token next ()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
54 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
55 switch (getNextChar)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
56 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
57 case CharType.EOF:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
58 Location l;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
59 return Token (Tok.EOF, l, 0);
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
60
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
61 case CharType.Whitespace:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
62 position += 1;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
63 return this.next;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
64
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
65 case CharType.Symbol:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
66 return lexSymbol;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
67
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
68 case CharType.Letter:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
69 return lexLetter;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
70
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
71 case CharType.Number:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
72 return lexNumber;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
73 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
74 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
75
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
76 Token peek ( int skip = 0)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
77 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
78 int oldPosition = this.position;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
79 while(skip-- > 0)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
80 this.next;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
81 Token t = this.next;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
82 this.position = oldPosition;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
83 return t;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
84 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
85
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
86 public Error[] getErrors()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
87 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
88 return this.errors;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
89 }
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
90
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
91 private:
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
92 Token eq()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
93 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
94 if(source.data[position] == '=')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
95 return Token(Tok.Eq, Location(position++ - 1, source), 2);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
96 return Token(Tok.Assign, Location(position - 1, source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
97 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
98 Token openBrace()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
99 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
100 return Token(Tok.OpenBrace, Location(position - 1, source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
101 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
102 Token openParentheses()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
103 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
104 return Token(Tok.OpenParentheses, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
105 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
106 Token closeParentheses()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
107 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
108 return Token(Tok.CloseParentheses, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
109 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
110 Token closeBrace()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
111 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
112 return Token(Tok.CloseBrace, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
113 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
114 Token seperator()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
115 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
116 Token t = Token(Tok.Seperator, Location(position - 1, source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
117 return t;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
118 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
119 Token colon()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
120 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
121 return Token(Tok.Colon, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
122 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
123 Token dot()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
124 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
125 return Token(Tok.Dot, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
126 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
127 Token comma()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
128 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
129 return Token(Tok.Comma, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
130 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
131 Token ne()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
132 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
133 if(source.data[position] == '=')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
134 return Token(Tok.Ne, Location(position++ - 1, this.source), 2);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
135 return Token(Tok.Not, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
136 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
137 Token le()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
138 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
139 if(source.data[position] == '=')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
140 return Token(Tok.Le, Location(position++ - 1, this.source), 2);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
141 return Token(Tok.Lt, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
142 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
143 Token ge()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
144 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
145 if(source.data[position] == '=')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
146 return Token(Tok.Ge, Location(position++ - 1, this.source), 2);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
147 return Token(Tok.Gt, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
148 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
149 Token add()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
150 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
151 return Token(Tok.Add, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
152 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
153 Token sub()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
154 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
155 return Token(Tok.Sub, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
156 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
157 Token mul()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
158 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
159 return Token(Tok.Mul, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
160 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
161 Token div()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
162 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
163 switch(source.data[position])
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
164 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
165 case '/':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
166 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
167 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
168 if(source.data[position++] == '\n')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
169 return this.next;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
170 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
171 return Token(Tok.EOF, Location(position, this.source), 0);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
172
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
173 case '*':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
174 position += 2;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
175 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
176 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
177 ++position;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
178 if(source.data[position-2] == '*')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
179 if(source.data[position-1] == '/')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
180 return this.next;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
181 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
182 throw error(__LINE__, "Unexpected end of file. Unclosed comment block");
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
183
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
184 case '+':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
185 position += 2;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
186 int nesting = 1;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
187 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
188 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
189 ++position;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
190 if(source.data[position-2] == '+')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
191 if(source.data[position-1] == '/')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
192 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
193 position++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
194 nesting--;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
195 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
196
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
197 if(source.data[position-2] == '/')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
198 if(source.data[position-1] == '+')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
199 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
200 nesting++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
201 position++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
202 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
203
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
204 if(nesting == 0)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
205 return this.next;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
206 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
207 throw error(__LINE__, "Unexpected end of file. Unclosed comment block");
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
208
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
209 default:
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
210 return Token(Tok.Div, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
211 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
212 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
213
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
214 Token lexNumber ()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
215 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
216 int i = 0;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
217 while(getNextChar(++i) == CharType.Number)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
218 {}
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
219
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
220 position += i;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
221
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
222 return Token(Tok.Integer, Location(position - i, this.source), i);
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
223 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
224
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
225 Token lexSymbol ()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
226 {
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
227 Token t = symbolFunctions[source.data[position++]]();
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
228
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
229 return t;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
230 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
231
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
232 Token lexLetter ()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
233 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
234 int i = 0;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
235 bool hasNumber = false;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
236 while (getNextChar(++i) == CharType.Letter ||
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
237 getNextChar(i) == CharType.Number)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
238 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
239 if (getNextChar(i) == CharType.Number)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
240 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
241 hasNumber = true;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
242 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
243 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
244
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
245 Token t = Token(Tok.Identifier, Location(position, source), i);
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
246
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
247 if (!hasNumber)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
248 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
249 char[] str = source.data[position .. position + i];
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
250 if(str in keywords)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
251 t.type = keywords[str];
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
252 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
253
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
254 position += i;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
255
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
256 return t;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
257 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
258
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
259 CharType getNextChar(int offset = 0)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
260 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
261 if (position + offset >= this.source.data.length)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
262 return CharType.EOF;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
263
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
264 char current = source.data[position + offset];
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
265
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
266 CharType c = charTable[current];
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
267
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
268 // if(c == CharType.INVALID)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
269 // throw error(__LINE__, "Read invalid symbol: '%0'").arg(current);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
270
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
271 return c;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
272
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
273 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
274
29
41d23f2762c3 Merge, and updated Error class
Anders Halager <halager@gmail.com>
parents: 28
diff changeset
275 Error error(uint line, char[] msg)
41d23f2762c3 Merge, and updated Error class
Anders Halager <halager@gmail.com>
parents: 28
diff changeset
276 {
41d23f2762c3 Merge, and updated Error class
Anders Halager <halager@gmail.com>
parents: 28
diff changeset
277 return (new Error(msg)).loc(Location(position, source));
41d23f2762c3 Merge, and updated Error class
Anders Halager <halager@gmail.com>
parents: 28
diff changeset
278 }
41d23f2762c3 Merge, and updated Error class
Anders Halager <halager@gmail.com>
parents: 28
diff changeset
279
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
280 int position;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
281 DataSource source;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
282 Error[] errors;
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
283 CharType[] charTable;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
284 Token delegate()[] symbolFunctions;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
285 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
286
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
287 enum CharType : ubyte
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
288 {
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
289 INVALID,
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
290 Letter,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
291 Number,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
292 Symbol,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
293 Whitespace,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
294
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
295 EOF
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
296 }