Mercurial > projects > dang

annotate src/lexer/Lexer.d @ 206:d3c148ca429b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Major moving of files. all src now goes into src, all docs in docs.
author Anders Johnsen <skabet@gmail.com>
date Tue, 12 Aug 2008 18:14:56 +0200
parents
children e0551773a005
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
206
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
1 module lexer.Lexer;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
2
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
3 import basic.Message,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
4 basic.SourceManager;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
5
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
6 import lexer.Token,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
7 lexer.Keyword;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
8
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
9 import tango.io.Stdout;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
10
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
11 /**
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
12 The Lexer class will supply you with methods to tokenize a D file. Supply the
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
13 Lexer with a DataSource and you can 'peek' and 'next' Tokens from the file.
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
14
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
15 For more info about Tokens, look up the lexer.Token module.
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
16 */
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
17 class Lexer
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
18 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
19 public:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
20
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
21 /**
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
22 Create a new Lexer.
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
23 */
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
24 this(SourceLocation start, SourceManager src_mgr, MessageHandler messages)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
25 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
26 this.messages = messages;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
27 sm = src_mgr;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
28 start_loc = start;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
29 position = 0;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
30 source = sm.getRawData(start_loc);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
31
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
32
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
33 charTable.length = 256;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
34 foreach (c; "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_")
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
35 charTable[c] = CharType.Letter;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
36
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
37 foreach (c; "0123456789")
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
38 charTable[c] = CharType.Number;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
39
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
40 foreach (c; "(){}[];:.,=!<>+-*/%\"`")
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
41 charTable[c] = CharType.Symbol;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
42
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
43 foreach (c; " \n")
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
44 charTable[c] = CharType.Whitespace;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
45
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
46 foreach (c; "'\\")
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
47 charTable[c] = CharType.Other;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
48
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
49 symbolFunctions.length = 256;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
50
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
51 symbolFunctions['('] = &openParentheses;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
52 symbolFunctions[')'] = &closeParentheses;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
53 symbolFunctions['{'] = &openBrace;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
54 symbolFunctions['}'] = &closeBrace;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
55 symbolFunctions['['] = &openBracket;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
56 symbolFunctions[']'] = &closeBracket;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
57 symbolFunctions[';'] = &seperator;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
58 symbolFunctions[':'] = &colon;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
59 symbolFunctions['.'] = &dot;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
60 symbolFunctions[','] = &comma;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
61 symbolFunctions['='] = &eq;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
62 symbolFunctions['!'] = &ne;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
63 symbolFunctions['<'] = &le;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
64 symbolFunctions['>'] = &ge;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
65 symbolFunctions['+'] = &plus;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
66 symbolFunctions['-'] = &minus;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
67 symbolFunctions['*'] = &star;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
68 symbolFunctions['/'] = &slash;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
69 symbolFunctions['%'] = &percent;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
70 symbolFunctions['"'] = &string;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
71 symbolFunctions['`'] = &string;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
72 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
73
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
74 /**
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
75 Get the next token from the source. This method will move the
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
76 internal position forward to the next Token.
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
77
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
78 return: A Token - Token.type is TokType.EOF if there is
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
79 no more tokens in the file.
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
80 */
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
81 Token next()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
82 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
83 switch (getNextChar)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
84 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
85 case CharType.EOF:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
86 SLoc loc;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
87 return Token(Tok.EOF, loc, 0);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
88
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
89 case CharType.Whitespace:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
90 position += 1;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
91 return this.next;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
92
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
93 case CharType.Symbol:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
94 return lexSymbol;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
95
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
96 case CharType.Letter:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
97 return lexLetter;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
98
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
99 case CharType.Number:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
100 return lexNumber;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
101 case CharType.Other:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
102 messages.report(UnexpectedTok, Loc(position)).fatal(ExitLevel.Lexer);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
103 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
104 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
105
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
106 /**
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
107 Get the next token from the source. This method will NOT move the
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
108 internal position forward, and thereby having no side-effects.
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
109
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
110 return: A Token - Token.type is TokType.EOF if there is
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
111 no more tokens in the file.
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
112 */
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
113 Token peek(int skip = 0)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
114 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
115 int oldPosition = this.position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
116 while (skip-- > 0)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
117 this.next;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
118 Token t = this.next;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
119 this.position = oldPosition;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
120 return t;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
121 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
122
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
123 private:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
124 Token eq()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
125 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
126 if(source[position] == '=')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
127 return Token(Tok.Eq, Loc(position++ - 1), 2);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
128 return Token(Tok.Assign, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
129 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
130 Token openBrace()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
131 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
132 return Token(Tok.OpenBrace, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
133 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
134 Token closeBrace()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
135 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
136 return Token(Tok.CloseBrace, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
137 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
138 Token openParentheses()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
139 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
140 return Token(Tok.OpenParentheses, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
141 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
142 Token closeParentheses()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
143 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
144 return Token(Tok.CloseParentheses, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
145 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
146 Token openBracket()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
147 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
148 return Token(Tok.OpenBracket, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
149 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
150 Token closeBracket()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
151 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
152 return Token(Tok.CloseBracket, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
153 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
154 Token seperator()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
155 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
156 return Token(Tok.Seperator, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
157 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
158 Token colon()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
159 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
160 return Token(Tok.Colon, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
161 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
162 Token dot()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
163 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
164 int pos = 0;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
165 while(getNextChar(0) == CharType.Number ||
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
166 this.source[position + pos + 1] == '_')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
167 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
168 if(getNextChar(0) == CharType.Number)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
169 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
170 position--;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
171 return lexNumber();
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
172 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
173 pos++;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
174 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
175 return Token(Tok.Dot, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
176 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
177 Token comma()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
178 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
179 return Token(Tok.Comma, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
180 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
181 Token ne()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
182 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
183 if(source[position] == '=')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
184 return Token(Tok.Ne, Loc(position++ - 1), 2);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
185 return Token(Tok.Not, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
186 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
187 Token le()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
188 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
189 if(source[position] == '=')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
190 return Token(Tok.Le, Loc(position++ - 1), 2);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
191 return Token(Tok.Lt, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
192 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
193 Token ge()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
194 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
195 if(source[position] == '=')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
196 return Token(Tok.Ge, Loc(position++ - 1), 2);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
197 return Token(Tok.Gt, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
198 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
199 Token plus()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
200 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
201 return Token(Tok.Plus, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
202 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
203 Token minus()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
204 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
205 return Token(Tok.Minus, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
206 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
207 Token star()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
208 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
209 return Token(Tok.Star, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
210 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
211 Token slash()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
212 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
213 switch(source[position])
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
214 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
215 case '/':
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
216 while(getNextChar != CharType.EOF)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
217 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
218 if(source[position++] == '\n')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
219 return this.next;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
220 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
221 return Token(Tok.EOF, Loc(position), 0);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
222
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
223 case '*':
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
224 position += 2;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
225 while(getNextChar != CharType.EOF)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
226 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
227 ++position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
228 if(source[position-2] == '*')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
229 if(source[position-1] == '/')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
230 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
231 return this.next;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
232 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
233 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
234 messages.report(UnexpectedEOFBlock,Loc(position));
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
235
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
236 case '+':
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
237 position += 2;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
238 int nesting = 1;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
239 while(getNextChar != CharType.EOF)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
240 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
241 ++position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
242 if(source[position-2] == '+')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
243 if(source[position-1] == '/')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
244 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
245 position++;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
246 nesting--;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
247 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
248
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
249 if(source[position-2] == '/')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
250 if(source[position-1] == '+')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
251 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
252 nesting++;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
253 position++;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
254 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
255
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
256 if(nesting == 0)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
257 return this.next;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
258 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
259 messages.report(UnexpectedEOFBlock,Loc(position));
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
260
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
261 default:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
262 return Token(Tok.Slash, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
263 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
264 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
265
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
266 Token percent()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
267 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
268 return Token(Tok.Percent, Loc(position - 1), 1);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
269 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
270
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
271 Token string()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
272 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
273 --position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
274 int start = position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
275 if(getNextChar() == CharType.Letter)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
276 position++;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
277 char end = '`';
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
278 switch(source[position])
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
279 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
280 case '"':
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
281 if(position > 0)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
282 if(source[position-1] == 'r')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
283 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
284 end = '"';
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
285 goto string_wys;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
286 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
287 ++position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
288 while(getNextChar != CharType.EOF)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
289 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
290 ++position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
291 if (source[position-1] == '"' )
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
292 return Token(Tok.String, Loc(start), position - start);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
293 else if (source[position-1] == '\\')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
294 position++;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
295 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
296 break;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
297 case '`':
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
298 string_wys:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
299 ++position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
300 while(getNextChar != CharType.EOF)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
301 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
302 ++position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
303 if (source[position-1] == end )
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
304 return Token(Tok.String, Loc(start), position - start);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
305 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
306 break;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
307 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
308 messages.report(UnexpectedEOFBlock, Loc(position)).fatal(ExitLevel.Lexer);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
309 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
310
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
311 Token lexNumber ()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
312 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
313 bool sign = false;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
314 bool dot = false;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
315 bool e = false;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
316
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
317 int i = 0;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
318
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
319 bool end = false;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
320 while(!end)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
321 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
322 switch(getNextChar(i))
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
323 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
324 case CharType.Number:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
325 break;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
326 case CharType.Symbol:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
327 if(this.source[position+i] == '.')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
328 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
329 if(dot)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
330 messages.report(OnlyOneDotFloating, Loc(position + i));
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
331 dot = true;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
332 break;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
333 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
334 end = true;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
335 continue;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
336 case CharType.Letter:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
337 if(this.source[position+i] == '_')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
338 break;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
339 if (this.source[position+i] == 'e' ||
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
340 this.source[position+i] == 'E')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
341 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
342 if (e)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
343 messages.report(OnlyOneEFloating, Loc(position + i));
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
344 e = true;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
345 break;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
346 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
347 end = true;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
348 continue;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
349
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
350 default:
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
351 end = true;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
352 continue;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
353 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
354 i++;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
355 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
356
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
357 position += i;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
358
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
359 return Token(Tok.Integer, Loc(position - i), i);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
360 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
361
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
362 Token lexSymbol ()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
363 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
364 Token t = symbolFunctions[source[position++]]();
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
365
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
366 return t;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
367 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
368
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
369 Token lexLetter ()
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
370 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
371 int i = 0;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
372 bool hasNumber = false;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
373 if (source[position+1] == '"' ||
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
374 source[position+1] == '`')
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
375 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
376 ++position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
377 return string;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
378 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
379 while (getNextChar(++i) == CharType.Letter ||
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
380 getNextChar(i) == CharType.Number)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
381 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
382 if (getNextChar(i) == CharType.Number)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
383 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
384 hasNumber = true;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
385 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
386 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
387
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
388 Token t = Token(Tok.Identifier, Loc(), i);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
389
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
390 if (!hasNumber)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
391 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
392 char[] str = source[position .. position + i];
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
393 if(str in keywords)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
394 t.type = keywords[str];
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
395 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
396
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
397 position += i;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
398
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
399 return t;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
400 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
401
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
402 CharType getNextChar(int offset = 0)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
403 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
404 if (position + offset >= this.source.length)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
405 return CharType.EOF;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
406
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
407 char current = source[position + offset];
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
408
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
409 CharType c = charTable[current];
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
410
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
411 if(c == CharType.INVALID)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
412 messages.report(InvalidSymbol, Loc())
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
413 .arg(Integer.toString(cast(int)current))
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
414 .fatal(ExitLevel.Lexer);
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
415
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
416 return c;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
417
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
418 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
419
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
420 private final SourceLocation Loc(int pos = -1)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
421 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
422 if (pos < 0)
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
423 return start_loc + position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
424 return start_loc + pos;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
425 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
426
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
427 SourceManager sm;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
428 SourceLocation start_loc;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
429 int position;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
430 char[] source;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
431 MessageHandler messages;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
432 CharType[] charTable;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
433 Token delegate()[] symbolFunctions;
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
434 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
435
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
436 enum CharType : ubyte
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
437 {
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
438 INVALID,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
439 Letter,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
440 Number,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
441 Symbol,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
442 Whitespace,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
443 Other,
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
444
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
445 EOF
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
446 }
d3c148ca429b Major moving of files. all src now goes into src, all docs in docs.
Anders Johnsen <skabet@gmail.com>
parents:
diff changeset
447