Mercurial > projects > dang

annotate src/lexer/Lexer.d @ 207:e0551773a005

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Added the correct version.
author Anders Johnsen <skabet@gmail.com>
date Tue, 12 Aug 2008 18:19:34 +0200
parents d3c148ca429b
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
1 module lexer.Lexer;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
2
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
3 import basic.Message,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
4 basic.SourceManager;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
5
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
6 import lexer.Token,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
7 lexer.Keyword;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
8
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
9 import tango.io.Stdout;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
10
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
11 /**
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
12 The Lexer class will supply you with methods to tokenize a D file. Supply the
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
13 Lexer with a DataSource and you can 'peek' and 'next' Tokens from the file.
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
14
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
15 For more info about Tokens, look up the lexer.Token module.
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
16 */
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
17 class Lexer
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
18 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
19 public:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
20
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
21 /**
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
22 Create a new Lexer.
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
23 */
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
24 this(SourceLocation start, SourceManager src_mgr, MessageHandler messages)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
25 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
26 this.messages = messages;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
27 sm = src_mgr;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
28 start_loc = start;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
29 position = 0;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
30 source = sm.getRawData(start_loc);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
31
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
32
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
33 charTable.length = 256;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
34 foreach (c; "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_")
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
35 charTable[c] = CharType.Letter;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
36
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
37 foreach (c; "0123456789")
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
38 charTable[c] = CharType.Number;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
39
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
40 foreach (c; "(){}[];:.,=!<>+-*/%&\"`")
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
41 charTable[c] = CharType.Symbol;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
42
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
43 foreach (c; " \n")
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
44 charTable[c] = CharType.Whitespace;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
45
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
46 foreach (c; "'\\")
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
47 charTable[c] = CharType.Other;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
48
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
49 symbolFunctions.length = 256;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
50
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
51 symbolFunctions['('] = &openParentheses;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
52 symbolFunctions[')'] = &closeParentheses;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
53 symbolFunctions['{'] = &openBrace;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
54 symbolFunctions['}'] = &closeBrace;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
55 symbolFunctions['['] = &openBracket;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
56 symbolFunctions[']'] = &closeBracket;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
57 symbolFunctions[';'] = &seperator;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
58 symbolFunctions[':'] = &colon;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
59 symbolFunctions['.'] = &dot;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
60 symbolFunctions[','] = &comma;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
61 symbolFunctions['='] = &eq;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
62 symbolFunctions['!'] = &ne;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
63 symbolFunctions['<'] = &le;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
64 symbolFunctions['>'] = &ge;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
65 symbolFunctions['+'] = &plus;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
66 symbolFunctions['-'] = &minus;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
67 symbolFunctions['*'] = &star;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
68 symbolFunctions['/'] = &slash;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
69 symbolFunctions['%'] = &percent;
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
70 symbolFunctions['&'] = &and;
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
71 symbolFunctions['"'] = &string;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
72 symbolFunctions['`'] = &string;
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
73
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
74 last = Token(Tok.EOF, SLoc() + 1, 0);
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
75 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
76
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
77 /**
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
78 Get the next token from the source. This method will move the
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
79 internal position forward to the next Token.
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
80
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
81 return: A Token - Token.type is TokType.EOF if there is
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
82 no more tokens in the file.
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
83 */
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
84 Token next()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
85 {
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
86 Token res;
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
87 switch (getNextChar)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
88 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
89 case CharType.EOF:
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
90 return Token(Tok.EOF, last.location, 0);
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
91
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
92 case CharType.Whitespace:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
93 position += 1;
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
94 res = this.next;
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
95 break;
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
96
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
97 case CharType.Symbol:
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
98 res = lexSymbol;
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
99 break;
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
100
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
101 case CharType.Letter:
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
102 res = lexLetter;
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
103 break;
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
104
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
105 case CharType.Number:
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
106 res = lexNumber;
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
107 break;
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
108 case CharType.Other:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
109 messages.report(UnexpectedTok, Loc(position)).fatal(ExitLevel.Lexer);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
110 }
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
111 if (res.type != Tok.EOF)
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
112 last = res;
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
113 return res;
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
114 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
115
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
116 /**
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
117 Get the next token from the source. This method will NOT move the
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
118 internal position forward, and thereby having no side-effects.
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
119
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
120 return: A Token - Token.type is TokType.EOF if there is
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
121 no more tokens in the file.
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
122 */
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
123 Token peek(int skip = 0)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
124 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
125 int oldPosition = this.position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
126 while (skip-- > 0)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
127 this.next;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
128 Token t = this.next;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
129 this.position = oldPosition;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
130 return t;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
131 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
132
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
133 Token last;
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
134 private:
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
135
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
136 Token eq()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
137 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
138 if(source[position] == '=')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
139 return Token(Tok.Eq, Loc(position++ - 1), 2);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
140 return Token(Tok.Assign, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
141 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
142 Token openBrace()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
143 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
144 return Token(Tok.OpenBrace, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
145 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
146 Token closeBrace()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
147 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
148 return Token(Tok.CloseBrace, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
149 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
150 Token openParentheses()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
151 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
152 return Token(Tok.OpenParentheses, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
153 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
154 Token closeParentheses()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
155 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
156 return Token(Tok.CloseParentheses, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
157 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
158 Token openBracket()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
159 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
160 return Token(Tok.OpenBracket, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
161 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
162 Token closeBracket()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
163 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
164 return Token(Tok.CloseBracket, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
165 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
166 Token seperator()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
167 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
168 return Token(Tok.Seperator, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
169 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
170 Token colon()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
171 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
172 return Token(Tok.Colon, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
173 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
174 Token dot()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
175 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
176 int pos = 0;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
177 while(getNextChar(0) == CharType.Number ||
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
178 this.source[position + pos + 1] == '_')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
179 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
180 if(getNextChar(0) == CharType.Number)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
181 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
182 position--;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
183 return lexNumber();
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
184 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
185 pos++;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
186 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
187 return Token(Tok.Dot, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
188 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
189 Token comma()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
190 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
191 return Token(Tok.Comma, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
192 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
193 Token ne()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
194 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
195 if(source[position] == '=')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
196 return Token(Tok.Ne, Loc(position++ - 1), 2);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
197 return Token(Tok.Not, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
198 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
199 Token le()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
200 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
201 if(source[position] == '=')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
202 return Token(Tok.Le, Loc(position++ - 1), 2);
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
203 if(source[position] == '<')
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
204 return Token(Tok.LeftShift, Loc(position++ - 1), 2);
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
205 return Token(Tok.Lt, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
206 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
207 Token ge()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
208 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
209 if(source[position] == '=')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
210 return Token(Tok.Ge, Loc(position++ - 1), 2);
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
211 if(source[position] == '>')
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
212 if(source[position+1] == '>')
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
213 {
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
214 position += 2;
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
215 return Token(Tok.UnsignedRightShift, Loc(position - 1), 3);
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
216 }
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
217 else
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
218 return Token(Tok.RightShift, Loc(position++ - 1), 2);
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
219 return Token(Tok.Gt, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
220 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
221 Token plus()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
222 {
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
223 if(source[position] == '=')
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
224 return Token(Tok.PlusAssign, Loc(position++ - 1), 2);
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
225 return Token(Tok.Plus, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
226 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
227 Token minus()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
228 {
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
229 if(source[position] == '=')
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
230 return Token(Tok.MinusAssign, Loc(position++ - 1), 2);
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
231 return Token(Tok.Minus, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
232 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
233 Token star()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
234 {
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
235 if(source[position] == '=')
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
236 return Token(Tok.StarAssign, Loc(position++ - 1), 2);
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
237 return Token(Tok.Star, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
238 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
239 Token slash()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
240 {
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
241 int p = position;
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
242 switch(source[position])
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
243 {
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
244 case '=':
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
245 return Token(Tok.SlashAssign, Loc(position++ - 1), 2);
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
246 case '/':
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
247 while(getNextChar != CharType.EOF)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
248 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
249 if(source[position++] == '\n')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
250 return this.next;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
251 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
252 return Token(Tok.EOF, Loc(position), 0);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
253
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
254 case '*':
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
255 position += 2;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
256 while(getNextChar != CharType.EOF)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
257 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
258 ++position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
259 if(source[position-2] == '*')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
260 if(source[position-1] == '/')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
261 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
262 return this.next;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
263 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
264 }
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
265 messages.report(UnexpectedEOFBlock,Loc(p)).fatal(ExitLevel.Lexer);
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
266
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
267 case '+':
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
268 position += 2;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
269 int nesting = 1;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
270 while(getNextChar != CharType.EOF)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
271 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
272 ++position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
273 if(source[position-2] == '+')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
274 if(source[position-1] == '/')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
275 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
276 position++;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
277 nesting--;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
278 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
279
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
280 if(source[position-2] == '/')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
281 if(source[position-1] == '+')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
282 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
283 nesting++;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
284 position++;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
285 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
286
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
287 if(nesting == 0)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
288 return this.next;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
289 }
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
290 messages.report(
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
291 UnexpectedEOFBlock,
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
292 Loc(p)).fatal(ExitLevel.Lexer);
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
293
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
294 default:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
295 return Token(Tok.Slash, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
296 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
297 }
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
298 Token and()
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
299 {
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
300 return Token(Tok.And, Loc(position - 1), 1);
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
301 }
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
302 Token percent()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
303 {
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
304 if(source[position] == '=')
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
305 return Token(Tok.PercentAssign, Loc(position++ - 1), 2);
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
306 return Token(Tok.Percent, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
307 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
308
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
309 Token string()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
310 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
311 --position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
312 int start = position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
313 if(getNextChar() == CharType.Letter)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
314 position++;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
315 char end = '`';
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
316 switch(source[position])
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
317 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
318 case '"':
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
319 if(position > 0)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
320 if(source[position-1] == 'r')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
321 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
322 end = '"';
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
323 goto string_wys;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
324 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
325 ++position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
326 while(getNextChar != CharType.EOF)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
327 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
328 ++position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
329 if (source[position-1] == '"' )
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
330 {
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
331 if(getNextChar != CharType.EOF)
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
332 if (source[position] == 'c' ||
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
333 source[position] == 'w' ||
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
334 source[position] == 'd')
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
335 position++;
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
336
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
337 return Token(Tok.String, Loc(start), position - start);
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
338 }
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
339 else if (source[position-1] == '\\')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
340 position++;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
341 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
342 break;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
343 case '`':
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
344 string_wys:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
345 ++position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
346 while(getNextChar != CharType.EOF)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
347 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
348 ++position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
349 if (source[position-1] == end )
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
350 return Token(Tok.String, Loc(start), position - start);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
351 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
352 break;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
353 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
354 messages.report(UnexpectedEOFBlock, Loc(position)).fatal(ExitLevel.Lexer);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
355 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
356
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
357 Token lexNumber ()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
358 {
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
359 bool sign;
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
360
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
361 int i = 0;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
362
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
363
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
364 bool end = false;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
365 while(!end)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
366 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
367 switch(getNextChar(i))
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
368 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
369 case CharType.Number:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
370 break;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
371 case CharType.Symbol:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
372 if(this.source[position+i] == '.')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
373 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
374 break;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
375 }
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
376 if (this.source[position+i] == '+' ||
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
377 this.source[position+i] == '-')
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
378 {
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
379 if (source[position+i-1] == 'e' ||
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
380 source[position+i-1] == 'E')
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
381 break;
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
382 }
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
383 end = true;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
384 continue;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
385 case CharType.Letter:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
386 if(this.source[position+i] == '_')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
387 break;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
388 if (this.source[position+i] == 'e' ||
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
389 this.source[position+i] == 'E')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
390 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
391 break;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
392 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
393 end = true;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
394 continue;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
395
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
396 default:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
397 end = true;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
398 continue;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
399 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
400 i++;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
401 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
402
207
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
403 while(source[position+i] == 'u' ||
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
404 source[position+i] == 'U' ||
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
405 source[position+i] == 'L')
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
406 i += 1;
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
407
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
408
e0551773a005 Added the correct version.
Anders Johnsen <skabet@gmail.com>
parents: 206
diff changeset
409
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
410 position += i;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
411
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
412 return Token(Tok.Integer, Loc(position - i), i);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
413 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
414
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
415 Token lexSymbol ()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
416 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
417 Token t = symbolFunctions[source[position++]]();
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
418
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
419 return t;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
420 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
421
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
422 Token lexLetter ()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
423 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
424 int i = 0;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
425 bool hasNumber = false;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
426 if (source[position+1] == '"' ||
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
427 source[position+1] == '`')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
428 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
429 ++position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
430 return string;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
431 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
432 while (getNextChar(++i) == CharType.Letter ||
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
433 getNextChar(i) == CharType.Number)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
434 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
435 if (getNextChar(i) == CharType.Number)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
436 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
437 hasNumber = true;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
438 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
439 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
440
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
441 Token t = Token(Tok.Identifier, Loc(), i);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
442
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
443 if (!hasNumber)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
444 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
445 char[] str = source[position .. position + i];
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
446 if(str in keywords)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
447 t.type = keywords[str];
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
448 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
449
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
450 position += i;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
451
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
452 return t;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
453 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
454
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
455 CharType getNextChar(int offset = 0)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
456 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
457 if (position + offset >= this.source.length)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
458 return CharType.EOF;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
459
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
460 char current = source[position + offset];
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
461
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
462 CharType c = charTable[current];
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
463
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
464 if(c == CharType.INVALID)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
465 messages.report(InvalidSymbol, Loc())
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
466 .arg(Integer.toString(cast(int)current))
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
467 .fatal(ExitLevel.Lexer);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
468
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
469 return c;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
470
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
471 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
472
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
473 private final SourceLocation Loc(int pos = -1)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
474 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
475 if (pos < 0)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
476 return start_loc + position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
477 return start_loc + pos;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
478 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
479
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
480 SourceManager sm;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
481 SourceLocation start_loc;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
482 int position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
483 char[] source;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
484 MessageHandler messages;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
485 CharType[] charTable;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
486 Token delegate()[] symbolFunctions;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
487 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
488
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
489 enum CharType : ubyte
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
490 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
491 INVALID,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
492 Letter,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
493 Number,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
494 Symbol,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
495 Whitespace,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
496 Other,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
497
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
498 EOF
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
499 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
500