projects/dang: lexer/Lexer.d annotate

annotate lexer/Lexer.d @ 72:628cb46ab13b new_gen

First update on the way to Arrays! :) Lexer and ast/Exp now have the types needed. All that needs to be updated now is the Parser. I suggest two options: 1) Making the Id a class that we can make a ArrayId and PointerId for at some point. 2) Use a struct as now, but adding some bools / ints to give a more abstract description of the type 2) is messy and 1) is slower... :/

author	Anders Johnsen <skabet@gmail.com>
date	Thu, 01 May 2008 23:08:41 +0200
parents	3fdf20b08a81
children	192da4976daa

rev	line source
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	1 module lexer.Lexer;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	2
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	3 import misc.Error,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	4 misc.DataSource;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	5
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	6 import lexer.Token,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	7 lexer.Keyword;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	8
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	9 import tango.io.Stdout;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	10
42 4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	11 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	12 The Lexer class will supply you with methods to tokenize a D file. Supply the
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	13 Lexer with a DataSource and you can 'peek' and 'next' Tokens from the file.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	14
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	15 For more info about Tokens, look up the lexer.Token module.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	16 */
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	17 class Lexer
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	18 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	19 public:
42 4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	20
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	21 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	22 Create a new Lexer.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	23
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	24 params:
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	25 source = The source to tokenize.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	26
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	27 */
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	28
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	29 this (DataSource source)
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	30 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	31 this.source = source;
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	32 position = 0;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	33
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	34
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	35 charTable.length = 256;
67 3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	36 foreach( char c ; "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_")
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	37 charTable[c] = CharType.Letter;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	38
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	39 foreach( char c ; "0123456789")
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	40 charTable[c] = CharType.Number;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	41
72 628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 67 diff changeset	42 foreach( char c ; "(){}[];:.,=!<>+-*/%")
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	43 charTable[c] = CharType.Symbol;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	44
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	45 foreach( char c ; " \n")
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	46 charTable[c] = CharType.Whitespace;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	47
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	48 symbolFunctions.length = 256;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	49
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	50 symbolFunctions['('] = &openParentheses;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	51 symbolFunctions[')'] = &closeParentheses;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	52 symbolFunctions['{'] = &openBrace;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	53 symbolFunctions['}'] = &closeBrace;
72 628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 67 diff changeset	54 symbolFunctions['['] = &openBracket;
628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 67 diff changeset	55 symbolFunctions[']'] = &closeBracket;
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	56 symbolFunctions[';'] = &seperator;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	57 symbolFunctions[':'] = &colon;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	58 symbolFunctions['.'] = &dot;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	59 symbolFunctions[','] = ,
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	60 symbolFunctions['='] = &eq;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	61 symbolFunctions['!'] = ≠
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	62 symbolFunctions['<'] = ≤
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	63 symbolFunctions['>'] = ≥
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	64 symbolFunctions['+'] = &add;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	65 symbolFunctions['-'] = ⊂
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	66 symbolFunctions['*'] = &mul;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	67 symbolFunctions['/'] = &div;
62 78a6808b2e0f Added support for "%" - modulus / reminder Anders Johnsen <skabet@gmail.com> parents: 43 diff changeset	68 symbolFunctions['%'] = &mod;
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	69 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	70
42 4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	71 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	72 Get the next token from the source. This method will move the
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	73 internal position forward to the next Token.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	74
43 a712c530b7cc A few fixes - now checking if chars are of a valid type Anders Johnsen <skabet@gmail.com> parents: 42 diff changeset	75 return: A Token - Token.type is TokType.EOF if there is
42 4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	76 no more tokens in the file.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	77 */
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	78 Token next ()
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	79 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	80 switch (getNextChar)
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	81 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	82 case CharType.EOF:
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	83 Location l;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	84 return Token (Tok.EOF, l, 0);
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	85
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	86 case CharType.Whitespace:
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	87 position += 1;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	88 return this.next;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	89
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	90 case CharType.Symbol:
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	91 return lexSymbol;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	92
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	93 case CharType.Letter:
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	94 return lexLetter;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	95
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	96 case CharType.Number:
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	97 return lexNumber;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	98 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	99 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	100
42 4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	101 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	102 Get the next token from the source. This method will NOT move the
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	103 internal position forward, and thereby having no side-effects.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	104
43 a712c530b7cc A few fixes - now checking if chars are of a valid type Anders Johnsen <skabet@gmail.com> parents: 42 diff changeset	105 return: A Token - Token.type is TokType.EOF if there is
42 4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	106 no more tokens in the file.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	107 */
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	108 Token peek ( int skip = 0)
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	109 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	110 int oldPosition = this.position;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	111 while(skip-- > 0)
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	112 this.next;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	113 Token t = this.next;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	114 this.position = oldPosition;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	115 return t;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	116 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	117
42 4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	118 /**
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	119 Return all errors that occurred while tokenizing the string.
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	120
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	121 TODO: Error system not implemented yet - this is a stub!
4e879f82dd64 Added some docs for the lexer - now you can understand _some_ of the madness going on here :) Anders Johnsen <skabet@gmail.com> parents: 41 diff changeset	122 */
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	123 public Error[] getErrors()
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	124 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	125 return this.errors;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	126 }
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	127
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	128 private:
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	129 Token eq()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	130 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	131 if(source.data[position] == '=')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	132 return Token(Tok.Eq, Location(position++ - 1, source), 2);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	133 return Token(Tok.Assign, Location(position - 1, source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	134 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	135 Token openBrace()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	136 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	137 return Token(Tok.OpenBrace, Location(position - 1, source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	138 }
72 628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 67 diff changeset	139 Token closeBrace()
628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 67 diff changeset	140 {
628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 67 diff changeset	141 return Token(Tok.CloseBrace, Location(position - 1, this.source), 1);
628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 67 diff changeset	142 }
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	143 Token openParentheses()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	144 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	145 return Token(Tok.OpenParentheses, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	146 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	147 Token closeParentheses()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	148 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	149 return Token(Tok.CloseParentheses, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	150 }
72 628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 67 diff changeset	151 Token openBracket()
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	152 {
72 628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 67 diff changeset	153 return Token(Tok.OpenBracket, Location(position - 1, this.source), 1);
628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 67 diff changeset	154 }
628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 67 diff changeset	155 Token closeBracket()
628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 67 diff changeset	156 {
628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 67 diff changeset	157 return Token(Tok.CloseBracket, Location(position - 1, source), 1);
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	158 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	159 Token seperator()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	160 {
72 628cb46ab13b First update on the way to Arrays! :) Anders Johnsen <skabet@gmail.com> parents: 67 diff changeset	161 return Token(Tok.Seperator, Location(position - 1, source), 1);
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	162 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	163 Token colon()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	164 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	165 return Token(Tok.Colon, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	166 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	167 Token dot()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	168 {
67 3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	169 int pos = 0;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	170 while(getNextChar(0) == CharType.Number \|\|
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	171 this.source.data[position + pos + 1] == '_')
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	172 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	173 if(getNextChar(0) == CharType.Number)
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	174 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	175 position--;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	176 return lexNumber();
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	177 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	178 pos++;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	179 }
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	180 return Token(Tok.Dot, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	181 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	182 Token comma()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	183 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	184 return Token(Tok.Comma, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	185 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	186 Token ne()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	187 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	188 if(source.data[position] == '=')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	189 return Token(Tok.Ne, Location(position++ - 1, this.source), 2);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	190 return Token(Tok.Not, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	191 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	192 Token le()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	193 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	194 if(source.data[position] == '=')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	195 return Token(Tok.Le, Location(position++ - 1, this.source), 2);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	196 return Token(Tok.Lt, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	197 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	198 Token ge()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	199 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	200 if(source.data[position] == '=')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	201 return Token(Tok.Ge, Location(position++ - 1, this.source), 2);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	202 return Token(Tok.Gt, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	203 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	204 Token add()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	205 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	206 return Token(Tok.Add, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	207 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	208 Token sub()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	209 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	210 return Token(Tok.Sub, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	211 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	212 Token mul()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	213 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	214 return Token(Tok.Mul, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	215 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	216 Token div()
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	217 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	218 switch(source.data[position])
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	219 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	220 case '/':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	221 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	222 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	223 if(source.data[position++] == '\n')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	224 return this.next;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	225 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	226 return Token(Tok.EOF, Location(position, this.source), 0);
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	227
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	228 case '*':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	229 position += 2;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	230 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	231 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	232 ++position;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	233 if(source.data[position-2] == '*')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	234 if(source.data[position-1] == '/')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	235 return this.next;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	236 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	237 throw error(__LINE__, "Unexpected end of file. Unclosed comment block");
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	238
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	239 case '+':
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	240 position += 2;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	241 int nesting = 1;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	242 while(getNextChar != CharType.EOF)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	243 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	244 ++position;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	245 if(source.data[position-2] == '+')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	246 if(source.data[position-1] == '/')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	247 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	248 position++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	249 nesting--;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	250 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	251
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	252 if(source.data[position-2] == '/')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	253 if(source.data[position-1] == '+')
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	254 {
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	255 nesting++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	256 position++;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	257 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	258
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	259 if(nesting == 0)
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	260 return this.next;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	261 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	262 throw error(__LINE__, "Unexpected end of file. Unclosed comment block");
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	263
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	264 default:
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	265 return Token(Tok.Div, Location(position - 1, this.source), 1);
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	266 }
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	267 }
62 78a6808b2e0f Added support for "%" - modulus / reminder Anders Johnsen <skabet@gmail.com> parents: 43 diff changeset	268
78a6808b2e0f Added support for "%" - modulus / reminder Anders Johnsen <skabet@gmail.com> parents: 43 diff changeset	269 Token mod()
78a6808b2e0f Added support for "%" - modulus / reminder Anders Johnsen <skabet@gmail.com> parents: 43 diff changeset	270 {
78a6808b2e0f Added support for "%" - modulus / reminder Anders Johnsen <skabet@gmail.com> parents: 43 diff changeset	271 return Token(Tok.Mod, Location(position - 1, this.source), 1);
78a6808b2e0f Added support for "%" - modulus / reminder Anders Johnsen <skabet@gmail.com> parents: 43 diff changeset	272 }
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	273
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	274 Token lexNumber ()
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	275 {
67 3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	276 bool sign = false;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	277 bool dot = false;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	278 bool e = false;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	279
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	280 int i = 0;
67 3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	281
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	282 bool end = false;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	283 while(!end)
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	284 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	285 switch(getNextChar(i))
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	286 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	287 case CharType.Number:
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	288 break;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	289 case CharType.Symbol:
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	290 if(this.source.data[position+i] == '.')
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	291 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	292 if(dot)
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	293 throw error(__LINE__,"Only one '.' is allowed in an floating number")
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	294 .tok(Token(Tok.Float, Location(position + i, this.source), 1));
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	295 dot = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	296 break;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	297 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	298 end = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	299 continue;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	300 case CharType.Letter:
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	301 if(this.source.data[position+i] == '_')
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	302 break;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	303 if (this.source.data[position+i] == 'e' \|\|
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	304 this.source.data[position+i] == 'E')
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	305 {
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	306 if (e)
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	307 throw error(__LINE__,"Only one '"~this.source.data[position+i]
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	308 ~"' is allowed in an floating number");
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	309 e = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	310 break;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	311 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	312 end = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	313 continue;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	314
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	315 default:
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	316 end = true;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	317 continue;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	318 }
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	319 i++;
3fdf20b08a81 Been working on the floating point parsing. Still a bit of work to do here. Anders Johnsen <skabet@gmail.com> parents: 62 diff changeset	320 }
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	321
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	322 position += i;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	323
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	324 return Token(Tok.Integer, Location(position - i, this.source), i);
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	325 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	326
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	327 Token lexSymbol ()
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	328 {
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	329 Token t = symbolFunctions[source.data[position++]]();
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	330
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	331 return t;
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	332 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	333
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	334 Token lexLetter ()
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	335 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	336 int i = 0;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	337 bool hasNumber = false;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	338 while (getNextChar(++i) == CharType.Letter \|\|
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	339 getNextChar(i) == CharType.Number)
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	340 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	341 if (getNextChar(i) == CharType.Number)
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	342 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	343 hasNumber = true;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	344 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	345 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	346
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	347 Token t = Token(Tok.Identifier, Location(position, source), i);
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	348
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	349 if (!hasNumber)
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	350 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	351 char[] str = source.data[position .. position + i];
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	352 if(str in keywords)
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	353 t.type = keywords[str];
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	354 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	355
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	356 position += i;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	357
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	358 return t;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	359 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	360
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	361 CharType getNextChar(int offset = 0)
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	362 {
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	363 if (position + offset >= this.source.data.length)
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	364 return CharType.EOF;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	365
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	366 char current = source.data[position + offset];
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	367
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	368 CharType c = charTable[current];
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	369
43 a712c530b7cc A few fixes - now checking if chars are of a valid type Anders Johnsen <skabet@gmail.com> parents: 42 diff changeset	370 if(c == CharType.INVALID)
a712c530b7cc A few fixes - now checking if chars are of a valid type Anders Johnsen <skabet@gmail.com> parents: 42 diff changeset	371 throw error(__LINE__, "Read invalid symbol: '%0'").arg(current);
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	372
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	373 return c;
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	374
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	375 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	376
29 41d23f2762c3 Merge, and updated Error class Anders Halager <halager@gmail.com> parents: 28 diff changeset	377 Error error(uint line, char[] msg)
41d23f2762c3 Merge, and updated Error class Anders Halager <halager@gmail.com> parents: 28 diff changeset	378 {
41d23f2762c3 Merge, and updated Error class Anders Halager <halager@gmail.com> parents: 28 diff changeset	379 return (new Error(msg)).loc(Location(position, source));
41d23f2762c3 Merge, and updated Error class Anders Halager <halager@gmail.com> parents: 28 diff changeset	380 }
41d23f2762c3 Merge, and updated Error class Anders Halager <halager@gmail.com> parents: 28 diff changeset	381
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	382 int position;
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	383 DataSource source;
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	384 Error[] errors;
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	385 CharType[] charTable;
f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	386 Token delegate()[] symbolFunctions;
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	387 }
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	388
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	389 enum CharType : ubyte
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	390 {
41 f977aa28eb32 Now using arrays insted of switch - should speed things up a notch! Anders Johnsen <skabet@gmail.com> parents: 36 diff changeset	391 INVALID,
1 2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	392 Letter,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	393 Number,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	394 Symbol,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	395 Whitespace,
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	396
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	397 EOF
2168f4cb73f1 First push johnsen@johnsen-desktop parents: diff changeset	398 }

Mercurial > projects > dang

annotate lexer/Lexer.d @ 72:628cb46ab13b new_gen