Mercurial > projects > dang

annotate lexer/Lexer.d @ 91:1a24e61eb104 new_gen

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fixed the SourceLocation/SourceManager for files stretching by more then one CheckPoints
author johnsen@johnsen-laptop
date Mon, 05 May 2008 17:07:16 +0200
parents a49bb982a7b0
children 771ac63898e2
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
1 module lexer.Lexer;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
2
89
a49bb982a7b0 Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents: 88
diff changeset
3 import basic.Message,
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
4 basic.SourceManager;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
5
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
6 import lexer.Token,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
7 lexer.Keyword;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
8
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
9 import tango.io.Stdout;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
10
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
11 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
12 The Lexer class will supply you with methods to tokenize a D file. Supply the
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
13 Lexer with a DataSource and you can 'peek' and 'next' Tokens from the file.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
14
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
15 For more info about Tokens, look up the lexer.Token module.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
16 */
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
17 class Lexer
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
18 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
19 public:
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
20
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
21 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
22 Create a new Lexer.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
23 */
89
a49bb982a7b0 Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents: 88
diff changeset
24
a49bb982a7b0 Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents: 88
diff changeset
25
a49bb982a7b0 Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents: 88
diff changeset
26 this(SourceLocation start, SourceManager src_mgr, MessageHandler messages)
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
27 {
89
a49bb982a7b0 Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents: 88
diff changeset
28 this.messages = messages;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
29 sm = src_mgr;
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
30 start_loc = start;
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
31 position = 0;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
32 source = sm.getRawData(start_loc);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
33
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
34
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
35 charTable.length = 256;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
36 foreach (c; "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_")
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
37 charTable[c] = CharType.Letter;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
38
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
39 foreach (c; "0123456789")
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
40 charTable[c] = CharType.Number;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
41
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
42 foreach (c; "(){}[];:.,=!<>+-*/%")
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
43 charTable[c] = CharType.Symbol;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
44
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
45 foreach (c; " \n")
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
46 charTable[c] = CharType.Whitespace;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
47
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
48 symbolFunctions.length = 256;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
49
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
50 symbolFunctions['('] = &openParentheses;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
51 symbolFunctions[')'] = &closeParentheses;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
52 symbolFunctions['{'] = &openBrace;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
53 symbolFunctions['}'] = &closeBrace;
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
54 symbolFunctions['['] = &openBracket;
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
55 symbolFunctions[']'] = &closeBracket;
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
56 symbolFunctions[';'] = &seperator;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
57 symbolFunctions[':'] = &colon;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
58 symbolFunctions['.'] = &dot;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
59 symbolFunctions[','] = &comma;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
60 symbolFunctions['='] = &eq;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
61 symbolFunctions['!'] = &ne;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
62 symbolFunctions['<'] = &le;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
63 symbolFunctions['>'] = &ge;
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
64 symbolFunctions['+'] = &plus;
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
65 symbolFunctions['-'] = &minus;
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
66 symbolFunctions['*'] = &star;
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
67 symbolFunctions['/'] = &slash;
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
68 symbolFunctions['%'] = &percent;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
69 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
70
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
71 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
72 Get the next token from the source. This method will move the
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
73 internal position forward to the next Token.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
74
43
a712c530b7cc A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents: 42
diff changeset
75 return: A Token - Token.type is TokType.EOF if there is
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
76 no more tokens in the file.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
77 */
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
78 Token next()
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
79 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
80 switch (getNextChar)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
81 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
82 case CharType.EOF:
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
83 SLoc loc;
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
84 return Token(Tok.EOF, loc, 0);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
85
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
86 case CharType.Whitespace:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
87 position += 1;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
88 return this.next;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
89
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
90 case CharType.Symbol:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
91 return lexSymbol;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
92
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
93 case CharType.Letter:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
94 return lexLetter;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
95
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
96 case CharType.Number:
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
97 return lexNumber;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
98 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
99 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
100
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
101 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
102 Get the next token from the source. This method will NOT move the
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
103 internal position forward, and thereby having no side-effects.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
104
43
a712c530b7cc A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents: 42
diff changeset
105 return: A Token - Token.type is TokType.EOF if there is
42
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
106 no more tokens in the file.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :)
Anders Johnsen <skabet@gmail.com>
parents: 41
diff changeset
107 */
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
108 Token peek(int skip = 0)
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
109 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
110 int oldPosition = this.position;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
111 while (skip-- > 0)
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
112 this.next;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
113 Token t = this.next;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
114 this.position = oldPosition;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
115 return t;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
116 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
117
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
118 private:
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
119 Token eq()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
120 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
121 if(source[position] == '=')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
122 return Token(Tok.Eq, Loc(position++ - 1), 2);
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
123 return Token(Tok.Assign, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
124 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
125 Token openBrace()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
126 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
127 return Token(Tok.OpenBrace, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
128 }
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
129 Token closeBrace()
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
130 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
131 return Token(Tok.CloseBrace, Loc(position - 1), 1);
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
132 }
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
133 Token openParentheses()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
134 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
135 return Token(Tok.OpenParentheses, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
136 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
137 Token closeParentheses()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
138 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
139 return Token(Tok.CloseParentheses, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
140 }
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
141 Token openBracket()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
142 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
143 return Token(Tok.OpenBracket, Loc(position - 1), 1);
72
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
144 }
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
145 Token closeBracket()
628cb46ab13b First update on the way to Arrays! :)
Anders Johnsen <skabet@gmail.com>
parents: 67
diff changeset
146 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
147 return Token(Tok.CloseBracket, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
148 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
149 Token seperator()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
150 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
151 return Token(Tok.Seperator, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
152 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
153 Token colon()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
154 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
155 return Token(Tok.Colon, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
156 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
157 Token dot()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
158 {
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
159 int pos = 0;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
160 while(getNextChar(0) == CharType.Number ||
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
161 this.source[position + pos + 1] == '_')
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
162 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
163 if(getNextChar(0) == CharType.Number)
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
164 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
165 position--;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
166 return lexNumber();
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
167 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
168 pos++;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
169 }
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
170 return Token(Tok.Dot, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
171 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
172 Token comma()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
173 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
174 return Token(Tok.Comma, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
175 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
176 Token ne()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
177 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
178 if(source[position] == '=')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
179 return Token(Tok.Ne, Loc(position++ - 1), 2);
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
180 return Token(Tok.Not, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
181 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
182 Token le()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
183 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
184 if(source[position] == '=')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
185 return Token(Tok.Le, Loc(position++ - 1), 2);
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
186 return Token(Tok.Lt, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
187 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
188 Token ge()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
189 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
190 if(source[position] == '=')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
191 return Token(Tok.Ge, Loc(position++ - 1), 2);
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
192 return Token(Tok.Gt, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
193 }
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
194 Token plus()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
195 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
196 return Token(Tok.Plus, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
197 }
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
198 Token minus()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
199 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
200 return Token(Tok.Minus, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
201 }
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
202 Token star()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
203 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
204 return Token(Tok.Star, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
205 }
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
206 Token slash()
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
207 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
208 switch(source[position])
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
209 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
210 case '/':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
211 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
212 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
213 if(source[position++] == '\n')
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
214 return this.next;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
215 }
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
216 return Token(Tok.EOF, Loc(position), 0);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
217
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
218 case '*':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
219 position += 2;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
220 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
221 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
222 ++position;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
223 if(source[position-2] == '*')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
224 if(source[position-1] == '/')
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
225 return this.next;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
226 }
89
a49bb982a7b0 Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents: 88
diff changeset
227 messages.report(UnexpectedEOFBlock,Loc(position));
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
228
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
229 case '+':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
230 position += 2;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
231 int nesting = 1;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
232 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
233 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
234 ++position;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
235 if(source[position-2] == '+')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
236 if(source[position-1] == '/')
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
237 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
238 position++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
239 nesting--;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
240 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
241
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
242 if(source[position-2] == '/')
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
243 if(source[position-1] == '+')
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
244 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
245 nesting++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
246 position++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
247 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
248
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
249 if(nesting == 0)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
250 return this.next;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
251 }
89
a49bb982a7b0 Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents: 88
diff changeset
252 messages.report(UnexpectedEOFBlock,Loc(position));
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
253
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
254 default:
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
255 return Token(Tok.Slash, Loc(position - 1), 1);
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
256 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
257 }
62
78a6808b2e0f Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents: 43
diff changeset
258
74
192da4976daa Renamed Add, Sub, Mul, Div and Mod in lexer to what they are (Plus, Minus....)
johnsen@johnsen-laptop
parents: 72
diff changeset
259 Token percent()
62
78a6808b2e0f Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents: 43
diff changeset
260 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
261 return Token(Tok.Percent, Loc(position - 1), 1);
62
78a6808b2e0f Added support for "%" - modulus / reminder
Anders Johnsen <skabet@gmail.com>
parents: 43
diff changeset
262 }
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
263
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
264 Token lexNumber ()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
265 {
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
266 bool sign = false;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
267 bool dot = false;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
268 bool e = false;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
269
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
270 int i = 0;
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
271
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
272 bool end = false;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
273 while(!end)
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
274 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
275 switch(getNextChar(i))
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
276 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
277 case CharType.Number:
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
278 break;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
279 case CharType.Symbol:
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
280 if(this.source[position+i] == '.')
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
281 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
282 if(dot)
89
a49bb982a7b0 Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents: 88
diff changeset
283 messages.report(OnlyOneDotFloating, Loc(position + i));
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
284 dot = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
285 break;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
286 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
287 end = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
288 continue;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
289 case CharType.Letter:
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
290 if(this.source[position+i] == '_')
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
291 break;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
292 if (this.source[position+i] == 'e' ||
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
293 this.source[position+i] == 'E')
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
294 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
295 if (e)
89
a49bb982a7b0 Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents: 88
diff changeset
296 messages.report(OnlyOneEFloating, Loc(position + i));
67
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
297 e = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
298 break;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
299 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
300 end = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
301 continue;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
302
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
303 default:
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
304 end = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
305 continue;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
306 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
307 i++;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here.
Anders Johnsen <skabet@gmail.com>
parents: 62
diff changeset
308 }
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
309
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
310 position += i;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
311
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
312 return Token(Tok.Integer, Loc(position - i), i);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
313 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
314
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
315 Token lexSymbol ()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
316 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
317 Token t = symbolFunctions[source[position++]]();
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
318
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
319 return t;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
320 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
321
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
322 Token lexLetter ()
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
323 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
324 int i = 0;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
325 bool hasNumber = false;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
326 while (getNextChar(++i) == CharType.Letter ||
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
327 getNextChar(i) == CharType.Number)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
328 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
329 if (getNextChar(i) == CharType.Number)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
330 {
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
331 hasNumber = true;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
332 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
333 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
334
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
335 Token t = Token(Tok.Identifier, Loc(), i);
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
336
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
337 if (!hasNumber)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
338 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
339 char[] str = source[position .. position + i];
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
340 if(str in keywords)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
341 t.type = keywords[str];
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
342 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
343
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
344 position += i;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
345
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
346 return t;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
347 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
348
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
349 CharType getNextChar(int offset = 0)
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
350 {
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
351 if (position + offset >= this.source.length)
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
352 return CharType.EOF;
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
353
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
354 char current = source[position + offset];
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
355
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
356 CharType c = charTable[current];
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
357
43
a712c530b7cc A few fixes - now checking if chars are of a valid type
Anders Johnsen <skabet@gmail.com>
parents: 42
diff changeset
358 if(c == CharType.INVALID)
91
1a24e61eb104 Fixed the SourceLocation/SourceManager for files stretching by more then one CheckPoints
johnsen@johnsen-laptop
parents: 89
diff changeset
359 messages.report(InvalidSymbol, Loc(), 1).arg(Integer.toString(cast(int)current));
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
360
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
361 return c;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
362
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
363 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
364
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
365 private final SourceLocation Loc(int pos = -1)
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
366 {
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
367 if (pos < 0)
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
368 return start_loc + position;
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
369 return start_loc + pos;
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
370 }
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
371
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
372 SourceManager sm;
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
373 SourceLocation start_loc;
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
374 int position;
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
375 char[] source;
89
a49bb982a7b0 Using the new SourceLocation system to handle errors. Also, this is the first push for making the errors don't throw, but continue to check the source.
Anders Johnsen <skabet@gmail.com>
parents: 88
diff changeset
376 MessageHandler messages;
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
377 CharType[] charTable;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
378 Token delegate()[] symbolFunctions;
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
379 }
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
380
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
381 enum CharType : ubyte
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
382 {
41
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch!
Anders Johnsen <skabet@gmail.com>
parents: 36
diff changeset
383 INVALID,
1
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
384 Letter,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
385 Number,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
386 Symbol,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
387 Whitespace,
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
388
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
389 EOF
2168f4cb73f1 First push
johnsen@johnsen-desktop
parents:
diff changeset
390 }
88
eb5b2c719a39 Major change to locations, tokens and expressions.
Anders Halager <halager@gmail.com>
parents: 74
diff changeset
391