projects/dil: trunk/src/Lexer.d annotate

annotate trunk/src/Lexer.d @ 67:996065105910

- Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. - Decoding utf-8 characters in scanHexStringLiteral().

author	aziz
date	Sun, 01 Jul 2007 10:58:03 +0000
parents	8e84db78ad55
children	7eb83dd38901

rev	line source
0 8ba2570de175 Initial import. aziz parents: diff changeset	1 /++
8ba2570de175 Initial import. aziz parents: diff changeset	2 Author: Aziz Köksal
8ba2570de175 Initial import. aziz parents: diff changeset	3 License: GPL2
8ba2570de175 Initial import. aziz parents: diff changeset	4 +/
8ba2570de175 Initial import. aziz parents: diff changeset	5 module Lexer;
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	6 import Token;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	7 import Keywords;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	8 import Identifier;
54 e55bd2270f94 - Relocated messages table to a separate module. aziz parents: 53 diff changeset	9 import Messages;
2 81c6cc33f5c8 - Initializing ptable with a precomputed array literal. aziz parents: 1 diff changeset	10 import std.stdio;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	11 import std.utf;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	12 import std.uni;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	13 import std.c.stdlib;
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	14 import std.stdarg;
66 8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	15 import std.string;
0 8ba2570de175 Initial import. aziz parents: diff changeset	16
11 dffcdaa7c47a - Added Unicode line and paragraph separators. aziz parents: 10 diff changeset	17 const char[3] LS = \u2028;
dffcdaa7c47a - Added Unicode line and paragraph separators. aziz parents: 10 diff changeset	18 const char[3] PS = \u2029;
dffcdaa7c47a - Added Unicode line and paragraph separators. aziz parents: 10 diff changeset	19
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	20 const dchar LSd = 0x2028;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	21 const dchar PSd = 0x2029;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	22
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	23 const uint _Z_ = 26; /// Control+Z
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	24
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	25 class Problem
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	26 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	27 enum Type
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	28 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	29 Lexer,
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	30 Parser,
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	31 Semantic
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	32 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	33
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	34 MID id;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	35 Type type;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	36 uint loc;
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	37 string[] arguments;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	38 /*
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	39 this(Type type, MID id, uint loc)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	40 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	41 this.id = id;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	42 this.type = type;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	43 this.loc = loc;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	44 }
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	45 */
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	46 this(Type type, MID id, uint loc, string[] arguments)
66 8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	47 {
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	48 this.id = id;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	49 this.type = type;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	50 this.loc = loc;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	51 this.arguments = arguments;
66 8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	52 }
8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	53
8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	54 string getMsg()
8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	55 {
8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	56 char[] msg = messages[id];
8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	57
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	58 if (arguments.length == 0)
66 8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	59 return msg;
8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	60
8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	61 foreach (i, arg; arguments)
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	62 msg = replace(msg, format("{%s}", i+1), arg);
66 8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	63
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	64 return msg;
66 8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	65 }
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	66 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	67
0 8ba2570de175 Initial import. aziz parents: diff changeset	68 class Lexer
8ba2570de175 Initial import. aziz parents: diff changeset	69 {
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	70 Token token;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	71 string text;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	72 char* p;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	73 char* end;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	74
17 9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	75 uint loc = 1; /// line of code
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	76
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	77 char[] fileName;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	78
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	79 Problem[] errors;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	80
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	81 Identifier[string] idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	82
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	83 this(string text, string fileName)
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	84 {
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	85 this.fileName = fileName;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	86
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	87 this.text = text;
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	88 if (text[$-1] != 0)
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	89 {
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	90 this.text.length = this.text.length + 1;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	91 this.text[$-1] = 0;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	92 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	93
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	94 this.p = this.text.ptr;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	95 this.end = this.p + this.text.length;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	96
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	97 loadKeywords();
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	98 }
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	99
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	100 public void scan(out Token t)
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	101 {
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	102 assert(p < end);
0 8ba2570de175 Initial import. aziz parents: diff changeset	103
10 3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	104 uint c = *p;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	105
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	106 while (1)
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	107 {
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	108 t.start = p;
16 476e8e55c1d4 - Added Whitespace to the character properties table. aziz parents: 15 diff changeset	109
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	110 if (c == 0)
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	111 {
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	112 assert(*p == 0);
17 9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	113 ++p;
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	114 assert(p == end);
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	115 t.type = TOK.EOF;
17 9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	116 t.end = p;
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	117 return;
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	118 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	119
17 9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	120 if (c == '\n')
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	121 {
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	122 c = *++p;
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	123 ++loc;
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	124 continue;
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	125 }
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	126 else if (c == '\r')
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	127 {
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	128 c = *++p;
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	129 if (c != '\n')
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	130 ++loc;
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	131 continue;
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	132 }
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	133 else if (c == LS[0] && p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2]))
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	134 {
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	135 p += 3;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	136 c = *p;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	137 continue;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	138 }
17 9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	139
13 e5211758b63c - Added isidbeg() function. aziz parents: 12 diff changeset	140 if (isidbeg(c))
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	141 {
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	142 if (c == 'r' && p[1] == '"' && ++p)
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	143 return scanRawStringLiteral(t);
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	144 if (c == 'x' && p[1] == '"')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	145 return scanHexStringLiteral(t);
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	146 Lidentifier:
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	147 do
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	148 { c = *++p; }
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	149 while (isident(c) \|\| c & 128 && isUniAlpha(decodeUTF8()))
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	150
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	151 t.end = p;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	152
65 6c21ae79fbb3 - Renamed function Token.span to Token.srcText. aziz parents: 64 diff changeset	153 string str = t.srcText;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	154 Identifier* id = str in idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	155
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	156 if (!id)
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	157 {
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	158 idtable[str] = Identifier.Identifier(TOK.Identifier, str);
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	159 id = str in idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	160 }
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	161 assert(id);
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	162 t.type = id.type;
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	163 return;
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	164 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	165
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	166 if (isdigit(c))
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	167 return scanNumber(t);
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	168
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	169 if (c == '/')
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	170 {
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	171 c = *++p;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	172 switch(c)
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	173 {
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	174 case '=':
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	175 ++p;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	176 t.type = TOK.DivAssign;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	177 t.end = p;
cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	178 return;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	179 case '+':
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	180 uint level = 1;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	181 while (1)
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	182 {
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	183 c = *++p;
42 c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	184 LswitchNC: // only jumped to from default case of next switch(c)
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	185 switch (c)
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	186 {
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	187 case '\r':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	188 if (p[1] == '\n')
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	189 ++p;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	190 case '\n':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	191 ++loc;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	192 continue;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	193 case 0, _Z_:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	194 error(MID.UnterminatedNestedComment);
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	195 goto LreturnNC;
42 c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	196 default:
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	197 }
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	198
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	199 c <<= 8;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	200 c \|= *++p;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	201 switch (c)
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	202 {
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	203 case 0x2F2B: // /+
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	204 ++level;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	205 continue;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	206 case 0x2B2F: // +/
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	207 if (--level == 0)
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	208 {
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	209 ++p;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	210 LreturnNC:
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	211 t.type = TOK.Comment;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	212 t.end = p;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	213 return;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	214 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	215 continue;
42 c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	216 case 0xE280: // LS[0..1] \|\| PS[0..1]
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	217 if (p[1] == LS[2] \|\| p[1] == PS[2])
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	218 {
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	219 ++loc;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	220 ++p;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	221 }
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	222 continue;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	223 default:
42 c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	224 c &= char.max;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	225 goto LswitchNC;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	226 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	227 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	228 case '*':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	229 while (1)
7 07e45c06a024 - Parsing nested comments correctly now. aziz parents: 5 diff changeset	230 {
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	231 c = *++p;
1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	232 LswitchBC: // only jumped to from default case of next switch(c)
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	233 switch (c)
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	234 {
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	235 case '\r':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	236 if (p[1] == '\n')
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	237 ++p;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	238 case '\n':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	239 ++loc;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	240 continue;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	241 case 0, _Z_:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	242 error(MID.UnterminatedBlockComment);
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	243 goto LreturnBC;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	244 default:
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	245 }
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	246
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	247 c <<= 8;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	248 c \|= *++p;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	249 switch (c)
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	250 {
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	251 case 0x2A2F: // */
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	252 ++p;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	253 LreturnBC:
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	254 t.type = TOK.Comment;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	255 t.end = p;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	256 return;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	257 case 0xE280: // LS[0..1] \|\| PS[0..1]
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	258 if (p[1] == LS[2] \|\| p[1] == PS[2])
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	259 {
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	260 ++loc;
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	261 ++p;
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	262 }
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	263 continue;
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	264 default:
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	265 c &= char.max;
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	266 goto LswitchBC;
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	267 }
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	268 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	269 assert(0);
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	270 case '/':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	271 while (1)
7 07e45c06a024 - Parsing nested comments correctly now. aziz parents: 5 diff changeset	272 {
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	273 c = *++p;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	274 switch (c)
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	275 {
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	276 case '\r':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	277 if (p[1] == '\n')
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	278 ++p;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	279 case '\n':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	280 case 0, _Z_:
11 dffcdaa7c47a - Added Unicode line and paragraph separators. aziz parents: 10 diff changeset	281 break;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	282 case LS[0]:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	283 if (p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2]))
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	284 break;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	285 continue;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	286 default:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	287 continue;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	288 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	289 t.type = TOK.Comment;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	290 t.end = p;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	291 return;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	292 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	293 default:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	294 t.type = TOK.Div;
10 3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	295 t.end = p;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	296 return;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	297 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	298 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	299
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	300 switch (c)
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	301 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	302 case '\'':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	303 return scanCharacterLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	304 case '`':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	305 return scanRawStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	306 case '"':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	307 return scanNormalStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	308 case '\\':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	309 char[] buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	310 do
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	311 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	312 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	313 c = scanEscapeSequence();
50 4a27b7840ea9 - Return error code 0xFFFF from scanEscapeSequence(). aziz parents: 49 diff changeset	314 if (c == 0xFFFF)
4a27b7840ea9 - Return error code 0xFFFF from scanEscapeSequence(). aziz parents: 49 diff changeset	315 break;
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	316 if (c < 128)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	317 buffer ~= c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	318 else
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	319 encodeUTF8(buffer, c);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	320 } while (*p == '\\')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	321 buffer ~= 0;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	322 t.type = TOK.String;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	323 t.str = buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	324 t.end = p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	325 return;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	326 case '>': /* > >= >> >>= >>> >>>= */
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	327 c = *++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	328 switch (c)
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	329 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	330 case '=':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	331 t.type = TOK.GreaterEqual;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	332 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	333 case '>':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	334 if (p[1] == '>')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	335 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	336 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	337 if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	338 { ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	339 t.type = TOK.URShiftAssign;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	340 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	341 else
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	342 t.type = TOK.URShift;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	343 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	344 else if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	345 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	346 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	347 t.type = TOK.RShiftAssign;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	348 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	349 else
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	350 t.type = TOK.RShift;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	351 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	352 default:
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	353 t.type = TOK.Greater;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	354 goto Lcommon2;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	355 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	356 assert(0);
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	357 case '<': /* < <= <> <>= << <<= */
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	358 c = *++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	359 switch (c)
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	360 {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	361 case '=':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	362 t.type = TOK.LessEqual;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	363 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	364 case '<':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	365 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	366 ++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	367 t.type = TOK.LShiftAssign;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	368 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	369 else
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	370 t.type = TOK.LShift;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	371 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	372 case '>':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	373 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	374 ++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	375 t.type = TOK.LorEorG;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	376 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	377 else
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	378 t.type = TOK.LorG;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	379 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	380 default:
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	381 t.type = TOK.Less;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	382 goto Lcommon2;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	383 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	384 assert(0);
37 7f3bcb97d017 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 36 diff changeset	385 case '!': /* ! !< !> !<= !>= !<> !<>= */
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	386 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	387 switch (c)
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	388 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	389 case '<':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	390 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	391 if (c == '>')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	392 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	393 if (p[1] == '=') {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	394 ++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	395 t.type = TOK.Unordered;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	396 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	397 else
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	398 t.type = TOK.UorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	399 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	400 else if (c == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	401 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	402 t.type = TOK.UorG;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	403 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	404 else {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	405 t.type = TOK.UorGorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	406 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	407 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	408 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	409 case '>':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	410 if (p[1] == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	411 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	412 ++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	413 t.type = TOK.UorL;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	414 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	415 else
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	416 t.type = TOK.UorLorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	417 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	418 case '=':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	419 t.type = TOK.NotEqual;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	420 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	421 default:
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	422 t.type = TOK.Not;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	423 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	424 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	425 assert(0);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	426 case '.': /* . .[0-9] .. ... */
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	427 if (p[1] == '.')
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	428 {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	429 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	430 if (p[1] == '.') {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	431 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	432 t.type = TOK.Ellipses;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	433 }
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	434 else
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	435 t.type = TOK.Slice;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	436 }
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	437 else if (isdigit(p[1]))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	438 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	439 return scanReal(t);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	440 }
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	441 else
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	442 t.type = TOK.Dot;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	443 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	444 case '\|': /* \| \|\| \|= */
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	445 c = *++p;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	446 if (c == '=')
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	447 t.type = TOK.OrAssign;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	448 else if (c == '\|')
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	449 t.type = TOK.OrLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	450 else {
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	451 t.type = TOK.OrBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	452 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	453 }
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	454 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	455 case '&': /* & && &= */
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	456 c = *++p;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	457 if (c == '=')
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	458 t.type = TOK.AndAssign;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	459 else if (c == '&')
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	460 t.type = TOK.AndLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	461 else {
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	462 t.type = TOK.AndBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	463 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	464 }
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	465 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	466 case '+': /* + ++ += */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	467 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	468 if (c == '=')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	469 t.type = TOK.PlusAssign;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	470 else if (c == '+')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	471 t.type = TOK.PlusPlus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	472 else {
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	473 t.type = TOK.Plus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	474 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	475 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	476 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	477 case '-': /* - -- -= */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	478 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	479 if (c == '=')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	480 t.type = TOK.MinusAssign;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	481 else if (c == '-')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	482 t.type = TOK.MinusMinus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	483 else {
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	484 t.type = TOK.Minus;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	485 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	486 }
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	487 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	488 case '=': /* = == */
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	489 if (p[1] == '=') {
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	490 ++p;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	491 t.type = TOK.Equal;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	492 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	493 else
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	494 t.type = TOK.Assign;
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	495 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	496 case '~': /* ~ ~= */
27 43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	497 if (p[1] == '=') {
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	498 ++p;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	499 t.type = TOK.CatAssign;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	500 }
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	501 else
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	502 t.type = TOK.Tilde;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	503 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	504 case '': / * = /
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	505 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	506 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	507 t.type = TOK.MulAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	508 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	509 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	510 t.type = TOK.Mul;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	511 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	512 case '^': /* ^ ^= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	513 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	514 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	515 t.type = TOK.XorAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	516 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	517 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	518 t.type = TOK.Xor;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	519 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	520 case '%': /* % %= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	521 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	522 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	523 t.type = TOK.ModAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	524 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	525 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	526 t.type = TOK.Mod;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	527 goto Lcommon;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	528 // Single character tokens:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	529 case '(':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	530 t.type = TOK.LParen;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	531 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	532 case ')':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	533 t.type = TOK.RParen;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	534 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	535 case '[':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	536 t.type = TOK.LBracket;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	537 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	538 case ']':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	539 t.type = TOK.RBracket;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	540 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	541 case '{':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	542 t.type = TOK.LBrace;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	543 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	544 case '}':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	545 t.type = TOK.RBrace;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	546 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	547 case ':':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	548 t.type = TOK.Colon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	549 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	550 case ';':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	551 t.type = TOK.Semicolon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	552 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	553 case '?':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	554 t.type = TOK.Question;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	555 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	556 case ',':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	557 t.type = TOK.Comma;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	558 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	559 case '$':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	560 t.type = TOK.Dollar;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	561 Lcommon:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	562 ++p;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	563 Lcommon2:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	564 t.end = p;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	565 return;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	566 case '#':
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	567 scanSpecialToken();
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	568 c = *p;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	569 continue;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	570 default:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	571 }
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	572
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	573 if (c & 128 && isUniAlpha(decodeUTF8()))
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	574 goto Lidentifier;
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	575 c = *++p;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	576 }
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	577 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	578
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	579 void peek(ref Token t)
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	580 {
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	581 char* tmp = p;
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	582 uint len = errors.length;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	583 scan(t);
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	584 p = tmp;
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	585 if (errors.length != len)
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	586 errors = errors[0..len];
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	587 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	588
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	589 void scanNormalStringLiteral(ref Token t)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	590 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	591 assert(*p == '"');
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	592 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	593 char[] buffer;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	594 t.type = TOK.String;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	595 while (1)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	596 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	597 switch (*p)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	598 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	599 case '"':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	600 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	601 Lreturn:
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	602 buffer ~= 0;
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	603 t.str = buffer;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	604 t.pf = scanPostfix();
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	605 t.end = p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	606 return;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	607 case '\\':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	608 ++p;
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	609 dchar d = scanEscapeSequence();
50 4a27b7840ea9 - Return error code 0xFFFF from scanEscapeSequence(). aziz parents: 49 diff changeset	610 if (d == 0xFFFF)
4a27b7840ea9 - Return error code 0xFFFF from scanEscapeSequence(). aziz parents: 49 diff changeset	611 continue;
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	612 if (d < 128)
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	613 buffer ~= d;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	614 else
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	615 encodeUTF8(buffer, d);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	616 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	617 case '\r':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	618 if (p[1] == '\n')
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	619 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	620 case '\n':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	621 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	622 ++loc;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	623 buffer ~= '\n';
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	624 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	625 case 0, _Z_:
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	626 error(MID.UnterminatedString);
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	627 goto Lreturn;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	628 default:
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	629 if (*p & 128)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	630 {
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	631 char* begin = p;
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	632 dchar d = decodeUTF8();
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	633 if (d == LSd \|\| d == PSd)
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	634 goto case '\n';
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	635
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	636 if (d != 0xFFFF)
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	637 {
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	638 ++p;
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	639 buffer ~= begin[0 .. p - begin];
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	640 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	641 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	642 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	643 buffer ~= *p++;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	644 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	645 }
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	646 assert(0);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	647 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	648
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	649 void scanCharacterLiteral(ref Token t)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	650 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	651 assert(*p == '\'');
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	652 MID id = MID.UnterminatedCharacterLiteral;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	653 ++p;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	654 switch (*p)
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	655 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	656 case '\\':
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	657 ++p;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	658 t.dchar_ = scanEscapeSequence();
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	659 break;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	660 case '\'':
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	661 ++p;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	662 id = MID.EmptyCharacterLiteral;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	663 case '\n', '\r', 0, _Z_:
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	664 goto Lerr;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	665 default:
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	666 uint c = *p;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	667 if (c & 128)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	668 {
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	669 c = decodeUTF8();
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	670 if (c == LSd \|\| c == PSd)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	671 goto Lerr;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	672 }
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	673 t.dchar_ = c;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	674 ++p;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	675 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	676
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	677 if (*p == '\'')
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	678 ++p;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	679 else
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	680 Lerr:
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	681 error(id);
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	682 t.type = TOK.Character;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	683 t.end = p;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	684 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	685
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	686 char scanPostfix()
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	687 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	688 switch (*p)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	689 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	690 case 'c':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	691 case 'w':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	692 case 'd':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	693 return *p++;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	694 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	695 return 0;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	696 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	697 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	698
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	699 void scanRawStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	700 {
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	701 uint delim = *p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	702 assert(delim == '`' \|\| delim == '"' && p[-1] == 'r');
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	703 t.type = TOK.String;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	704 char[] buffer;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	705 uint c;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	706 while (1)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	707 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	708 c = *++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	709 switch (c)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	710 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	711 case '\r':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	712 if (p[1] == '\n')
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	713 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	714 c = '\n'; // Convert '\r' and '\r\n' to '\n'
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	715 case '\n':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	716 ++loc;
52 f65a83c27638 - Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well. aziz parents: 51 diff changeset	717 break;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	718 case '`':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	719 case '"':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	720 if (c == delim)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	721 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	722 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	723 t.pf = scanPostfix();
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	724 Lreturn:
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	725 t.str = buffer ~ '\0';
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	726 t.end = p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	727 return;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	728 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	729 break;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	730 case LS[0]:
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	731 if (p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2]))
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	732 {
52 f65a83c27638 - Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well. aziz parents: 51 diff changeset	733 c = '\n';
f65a83c27638 - Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well. aziz parents: 51 diff changeset	734 ++p; ++p;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	735 ++loc;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	736 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	737 break;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	738 case 0, _Z_:
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	739 if (delim == 'r')
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	740 error(MID.UnterminatedRawString);
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	741 else
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	742 error(MID.UnterminatedBackQuoteString);
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	743 goto Lreturn;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	744 default:
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	745 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	746 buffer ~= c; // copy character to buffer
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	747 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	748 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	749 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	750
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	751 void scanHexStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	752 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	753 assert(p[0] == 'x' && p[1] == '"');
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	754 t.type = TOK.String;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	755
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	756 uint c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	757 ubyte[] buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	758 ubyte h; // hex number
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	759 uint n; // number of hex digits
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	760
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	761 ++p;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	762 while (1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	763 {
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	764 c = *++p;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	765 switch (c)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	766 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	767 case '"':
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	768 ++p;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	769 if (n & 1)
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	770 error(MID.OddNumberOfDigitsInHexString);
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	771 t.pf = scanPostfix();
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	772 Lreturn:
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	773 buffer ~= 0;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	774 t.str = cast(string) buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	775 t.end = p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	776 return;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	777 case '\r':
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	778 if (p[1] == '\n')
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	779 ++p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	780 case '\n':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	781 ++loc;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	782 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	783 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	784 if (ishexad(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	785 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	786 if (c <= '9')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	787 c -= '0';
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	788 else if (c <= 'F')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	789 c -= 'A' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	790 else
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	791 c -= 'a' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	792
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	793 if (n & 1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	794 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	795 h <<= 4;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	796 h \|= c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	797 buffer ~= h;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	798 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	799 else
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	800 h = c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	801 ++n;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	802 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	803 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	804 else if (isspace(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	805 continue;
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	806
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	807 if (c >= 128)
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	808 {
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	809 c = decodeUTF8();
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	810 if (c == LSd \|\| c == PSd)
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	811 {
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	812 ++p; ++p;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	813 ++loc;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	814 continue;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	815 }
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	816 }
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	817 else if (c == 0 \|\| c == _Z_)
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	818 {
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	819 error(MID.UnterminatedHexString);
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	820 t.pf = 0;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	821 goto Lreturn;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	822 }
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	823 error(MID.NonHexCharInHexString, cast(dchar)c);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	824 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	825 }
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	826 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	827 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	828
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	829 dchar scanEscapeSequence()
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	830 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	831 uint c = char2ev(*p);
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	832 if (c) {
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	833 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	834 return c;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	835 }
50 4a27b7840ea9 - Return error code 0xFFFF from scanEscapeSequence(). aziz parents: 49 diff changeset	836 c = 0xFFFF;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	837 uint digits = 2;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	838
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	839 switch (*p)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	840 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	841 case 'x':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	842 c = 0;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	843 while (1)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	844 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	845 ++p;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	846 if (ishexad(*p))
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	847 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	848 c *= 16;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	849 if (*p <= '9')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	850 c += *p - '0';
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	851 else if (*p <= 'F')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	852 c += *p - 'A' + 10;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	853 else
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	854 c += *p - 'a' + 10;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	855 if (!--digits) {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	856 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	857 break;
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	858 }
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	859 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	860 else
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	861 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	862 error(MID.InsufficientHexDigits);
50 4a27b7840ea9 - Return error code 0xFFFF from scanEscapeSequence(). aziz parents: 49 diff changeset	863 c = 0xFFFF;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	864 break;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	865 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	866 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	867 break;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	868 case 'u':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	869 digits = 4;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	870 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	871 case 'U':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	872 digits = 8;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	873 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	874 default:
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	875 if (isoctal(*p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	876 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	877 c = 0;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	878 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	879 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	880 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	881 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	882 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	883 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	884 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	885 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	886 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	887 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	888 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	889 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	890 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	891 else if(*p == '&')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	892 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	893 if (isalpha(*++p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	894 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	895 while (1)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	896 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	897 if (isalnum(*++p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	898 continue;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	899 if (*p == ';') {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	900 // TODO: convert entity to unicode codepoint.
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	901 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	902 break;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	903 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	904 else {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	905 error(MID.UnterminatedHTMLEntity);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	906 break;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	907 }
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	908 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	909 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	910 else
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	911 error(MID.InvalidBeginHTMLEntity);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	912 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	913 else
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	914 error(MID.UndefinedEscapeSequence);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	915 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	916
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	917 return c;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	918 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	919
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	920 /*
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	921 IntegerLiteral:= (Dec\|Hex\|Bin\|Oct)Suffix?
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	922 Dec:= (0\|[1-9][0-9_]*)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	923 Hex:= 0[xX] HexDigits
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	924 Bin:= 0[bB][01_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	925 Oct:= 0[0-7_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	926 Suffix:= (L\|[uU]\|L[uU]\|[uU]L)?
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	927 HexDigits:= [0-9a-zA-Z_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	928
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	929 Invalid: "0b_", "0x_", "._"
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	930 */
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	931 void scanNumber(ref Token t)
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	932 {
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	933 ulong ulong_;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	934 bool overflow;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	935 bool isDecimal;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	936 size_t digits;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	937
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	938 if (*p != '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	939 goto LscanInteger;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	940 ++p; // skip zero
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	941 // check for xX bB ...
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	942 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	943 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	944 case 'x','X':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	945 goto LscanHex;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	946 case 'b','B':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	947 goto LscanBin;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	948 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	949 if (p[1] == 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	950 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	951 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	952 if (p[1] == '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	953 break;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	954 case 'i','f','F', 'e', 'E': // Imaginary and float literal suffix
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	955 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	956 case '_':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	957 ++p;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	958 goto LscanOct;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	959 default:
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	960 if (isoctal(*p))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	961 goto LscanOct;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	962 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	963
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	964 ulong_ = p[-1];
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	965 isDecimal = true;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	966 goto Lfinalize;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	967
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	968 LscanInteger:
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	969 isDecimal = true;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	970 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	971 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	972 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	973 continue;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	974 if (!isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	975 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	976 if (ulong_ < ulong.max/10 \|\| (ulong_ == ulong.max/10 && *p <= '5'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	977 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	978 ulong_ *= 10;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	979 ulong_ += *p - '0';
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	980 continue;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	981 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	982 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	983 overflow = true;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	984 while (isdigit(*++p)) {}
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	985 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	986 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	987
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	988 // The number could be a float, so check overflow below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	989 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	990 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	991 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	992 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	993 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	994 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	995 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	996 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	997 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	998 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	999 goto LscanReal;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1000 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1001 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1002
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1003 if (overflow)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1004 error(MID.OverflowDecimalNumber);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1005
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1006 assert((isdigit(p[-1]) \|\| p[-1] == '_') && !isdigit(p) && p != '_');
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1007 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1008
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1009 LscanHex:
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1010 assert(digits == 0);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1011 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1012 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1013 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1014 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1015 if (!ishexad(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1016 break;
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1017 ++digits;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1018 ulong_ *= 16;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1019 if (*p <= '9')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1020 ulong_ += *p - '0';
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1021 else if (*p <= 'F')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1022 ulong_ += *p - 'A' + 10;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1023 else
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1024 ulong_ += *p - 'a' + 10;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1025 }
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1026
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1027 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1028 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1029 case '.':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1030 if (p[1] != '.')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1031 goto LscanHexReal;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1032 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1033 case 'L':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1034 if (p[1] != 'i')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1035 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1036 case 'i', 'p', 'P':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1037 goto LscanHexReal;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1038 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1039 }
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1040 if (digits == 0)
59 3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	1041 error(MID.NoDigitsInHexNumber);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1042 else if (digits > 16)
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1043 {
512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1044 // Overflow: skip following digits.
512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1045 error(MID.OverflowHexNumber);
512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1046 while (ishexad(*++p)) {}
512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1047 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1048 goto Lfinalize;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1049 LscanHexReal:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1050 return scanHexReal(t);
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1051
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1052 LscanBin:
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1053 assert(digits == 0);
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1054 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1055 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1056 if (*++p == '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1057 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1058 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1059 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1060 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1061 if (*p == '1')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1062 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1063 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1064 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1065 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1066 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1067 if (*p == '_')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1068 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1069 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1070 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1071
59 3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	1072 if (digits == 0)
3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	1073 error(MID.NoDigitsInBinNumber);
3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	1074
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1075 if (digits > 64)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1076 error(MID.OverflowBinaryNumber);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1077 assert((p[-1] == '0' \|\| p[-1] == '1' \|\| p[-1] == '_') && !(p == '0' \|\| p == '1' \|\| *p == '_'));
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1078 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1079
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1080 LscanOct:
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1081 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1082 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1083 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1084 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1085 if (!isoctal(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1086 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1087 if (ulong_ < ulong.max/2 \|\| (ulong_ == ulong.max/2 && *p <= '1'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1088 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1089 ulong_ *= 8;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1090 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1091 ++p;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1092 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1093 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1094 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1095 overflow = true;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1096 while (isdigit(*++p)) {}
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1097 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1098 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1099
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1100 bool hasDecimalDigits;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1101 if (isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1102 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1103 hasDecimalDigits = true;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1104 while (isdigit(*++p)) {}
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1105 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1106
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1107 // The number could be a float, so check errors below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1108 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1109 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1110 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1111 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1112 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1113 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1114 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1115 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1116 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1117 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1118 goto LscanReal;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1119 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1120 }
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1121
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1122 if (hasDecimalDigits)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1123 error(MID.OctalNumberHasDecimals);
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1124 if (overflow)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1125 error(MID.OverflowOctalNumber);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1126 // goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1127
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1128 Lfinalize:
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1129 enum Suffix
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1130 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1131 None = 0,
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1132 Unsigned = 1,
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1133 Long = 2
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1134 }
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1135
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1136 Suffix suffix;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1137 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1138 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1139 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1140 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1141 case 'L':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1142 if (suffix & Suffix.Long)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1143 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1144 suffix \|= Suffix.Long;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1145 ++p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1146 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1147 case 'u', 'U':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1148 if (suffix & Suffix.Unsigned)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1149 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1150 suffix \|= Suffix.Unsigned;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1151 ++p;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1152 continue;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1153 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1154 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1155 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1156 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1157 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1158
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1159 switch (suffix)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1160 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1161 case Suffix.None:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1162 if (ulong_ & 0x8000000000000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1163 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1164 if (isDecimal)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1165 error(MID.OverflowDecimalSign);
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1166 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1167 }
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1168 else if (ulong_ & 0xFFFFFFFF00000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1169 t.type = TOK.Int64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1170 else if (ulong_ & 0x80000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1171 t.type = isDecimal ? TOK.Int64 : TOK.Uint32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1172 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1173 t.type = TOK.Int32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1174 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1175 case Suffix.Unsigned:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1176 if (ulong_ & 0xFFFFFFFF00000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1177 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1178 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1179 t.type = TOK.Uint32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1180 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1181 case Suffix.Long:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1182 if (ulong_ & 0x8000000000000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1183 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1184 if (isDecimal)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1185 error(MID.OverflowDecimalSign);
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1186 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1187 }
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1188 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1189 t.type = TOK.Int64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1190 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1191 case Suffix.Unsigned \| Suffix.Long:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1192 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1193 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1194 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1195 assert(0);
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1196 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1197 t.ulong_ = ulong_;
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1198 t.end = p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1199 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1200 LscanReal:
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1201 scanReal(t);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1202 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1203 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1204
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1205 /*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1206 FloatLiteral:= Float[fFL]?i?
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1207 Float:= DecFloat \| HexFloat
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1208 DecFloat:= ([0-9][0-9_][.][0-9_]DecExponent?) \| [.][0-9][0-9_]DecExponent? \| [0-9][0-9_]DecExponent
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1209 DecExponent:= [eE][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1210 HexFloat:= 0[xX](HexDigits[.]HexDigits \| [.][0-9a-zA-Z]HexDigits? \| HexDigits)HexExponent
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1211 HexExponent:= [pP][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1212 */
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1213 void scanReal(ref Token t)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1214 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1215 if (*p == '.')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1216 // This function was called by scan() or scanNumber().
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1217 while (isdigit(++p) \|\| p == '_') {}
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1218 else
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1219 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1220 // This function was called by scanNumber().
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1221 debug switch (*p)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1222 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1223 case 'L':
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1224 if (p[1] != 'i')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1225 assert(0);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1226 case 'i', 'f', 'F', 'e', 'E': break;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1227 default: assert(0);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1228 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1229 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1230
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1231 // Scan exponent.
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1232 if (p == 'e' \|\| p == 'E')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1233 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1234 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1235 if (p == '-' \|\| p == '+')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1236 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1237 if (!isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1238 error(MID.FloatExponentDigitExpected);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1239 else
64 dd4c5a0e47dd - Improved while loop. aziz parents: 63 diff changeset	1240 while (isdigit(++p) \|\| p == '_') {}
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1241 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1242
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1243 // Copy string to buffer ignoring underscores.
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1244 char[] buffer;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1245 char* end = p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1246 p = t.start;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1247 do
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1248 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1249 if (*p == '_')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1250 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1251 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1252 continue;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1253 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1254 buffer ~= *p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1255 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1256 } while (p != end)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1257 buffer ~= 0;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1258 finalizeFloat(t, buffer);
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1259 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1260
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1261 void scanHexReal(ref Token t)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1262 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1263 assert(p == '.' \|\| p == 'i' \|\| p == 'p' \|\| p == 'P' \|\| (*p == 'L' && p[1] == 'i'));
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1264 MID mid;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1265 if (*p == '.')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1266 while (ishexad(++p) \|\| p == '_') {}
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1267 if (p != 'p' && p != 'P')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1268 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1269 mid = MID.HexFloatExponentRequired;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1270 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1271 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1272 // Copy mantissa to a buffer ignoring underscores.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1273 char* end = p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1274 p = t.start;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1275 char[] buffer;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1276 do
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1277 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1278 if (*p == '_')
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1279 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1280 ++p;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1281 continue;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1282 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1283 buffer ~= *p;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1284 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1285 } while (p != end)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1286
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1287 assert(p == end && (p == 'p' \|\| p == 'P'));
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1288 // Scan and copy the exponent.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1289 buffer ~= 'p';
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1290 size_t bufflen = buffer.length;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1291 while (1)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1292 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1293 if (*++p == '_')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1294 continue;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1295 if (isdigit(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1296 buffer ~= *p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1297 else
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1298 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1299 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1300 // When the buffer length hasn't changed, no digits were copied.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1301 if (bufflen == buffer.length) {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1302 mid = MID.HexFloatMissingExpDigits;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1303 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1304 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1305 buffer ~= 0; // Terminate for C functions.
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1306 finalizeFloat(t, buffer);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1307 return;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1308 Lerr:
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1309 t.type = TOK.Float32;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1310 t.end = p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1311 error(mid);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1312 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1313
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1314 void finalizeFloat(ref Token t, string buffer)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1315 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1316 // Float number is well-formed. Check suffixes and do conversion.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1317 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1318 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1319 case 'f', 'F':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1320 t.type = TOK.Float32;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1321 t.float_ = strtof(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1322 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1323 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1324 case 'L':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1325 t.type = TOK.Float80;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1326 t.real_ = strtold(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1327 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1328 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1329 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1330 t.type = TOK.Float64;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1331 t.double_ = strtod(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1332 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1333 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1334 if (*p == 'i')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1335 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1336 ++p;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1337 t.type += 3; // Switch to imaginary counterpart.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1338 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1339 if (getErrno == ERANGE)
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1340 error(MID.OverflowFloatNumber);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1341 t.end = p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1342 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1343
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1344 /// Scan special token: #line Integer [Filespec] EndOfLine
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1345 // TODO: Handle case like: #line 0 #line 2
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1346 void scanSpecialToken()
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1347 {
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1348 assert(*p == '#');
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1349
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1350 ++p;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1351 MID mid;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1352 Token t;
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1353 uint oldloc = this.loc, newloc;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1354
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1355 peek(t);
65 6c21ae79fbb3 - Renamed function Token.span to Token.srcText. aziz parents: 64 diff changeset	1356 if (!(this.loc == oldloc && p == t.start && t.type == TOK.Identifier && t.srcText == "line"))
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1357 {
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1358 this.loc = oldloc; // reset this.loc because we took a peek at the next token
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1359 mid = MID.ExpectedIdentifierSTLine;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1360 goto Lerr;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1361 }
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1362 p = t.end; // consume token
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1363
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1364 peek(t);
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1365 if (this.loc == oldloc && t.type == TOK.Number)
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1366 {
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1367 newloc = t.uint_ - 1;
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1368 p = t.end;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1369 }
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1370 else
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1371 {
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1372 this.loc = oldloc;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1373 mid = MID.ExpectedNumberAfterSTLine;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1374 goto Lerr;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1375 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1376
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1377 peek(t);
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1378 if (this.loc != oldloc)
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1379 {
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1380 this.loc = oldloc;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1381 mid = MID.NewlineInSpecialToken;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1382 goto Lerr;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1383 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1384 if (t.type == TOK.String)
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1385 {
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1386 if (*t.start != '"')
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1387 {
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1388 mid = MID.ExpectedNormalStringLiteral;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1389 goto Lerr;
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1390 }
65 6c21ae79fbb3 - Renamed function Token.span to Token.srcText. aziz parents: 64 diff changeset	1391 fileName = t.srcText[1..$-1]; // contents of "..."
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1392 p = t.end;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1393 }
65 6c21ae79fbb3 - Renamed function Token.span to Token.srcText. aziz parents: 64 diff changeset	1394 else if (t.type == TOK.Identifier && t.srcText == "__FILE__")
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1395 {
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1396 p = t.end;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1397 }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1398 /+
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1399 peek(t);
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1400 if (this.loc == oldloc && t.type != TOK.EOF)
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1401 {
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1402 mid = MID.UnterminatedSpecialToken;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1403 goto Lerr;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1404 }
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1405 +/
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1406 while (1)
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1407 {
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1408 switch (*p)
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1409 {
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1410 case '\r':
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1411 if (p[1] == '\n')
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1412 ++p;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1413 case '\n':
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1414 ++p;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1415 break;
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1416 case LS[0]:
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1417 if (p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2]))
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1418 {
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1419 p += 2;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1420 break;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1421 }
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1422 case 0, _Z_:
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1423 break;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1424 default:
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1425 if (isspace(*p)) {
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1426 ++p;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1427 continue;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1428 }
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1429 mid = MID.UnterminatedSpecialToken;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1430 goto Lerr;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1431 }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1432 break;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1433 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1434
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1435 this.loc = newloc;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1436 return;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1437 Lerr:
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1438 error(mid);
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1439 }
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1440
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1441 dchar decodeUTF8()
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1442 {
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1443 assert(*p & 128, "check for ASCII char before calling decodeUTF8().");
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1444 size_t idx;
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1445 uint d = 0xFFFF;
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1446 try
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1447 {
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1448 d = std.utf.decode(p[0 .. end-p], idx);
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1449 p += idx -1;
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1450 }
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1451 catch (UtfException e)
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1452 {
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1453 error(MID.InvalidUTF8Sequence);
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1454 // Skip to next valid utf-8 sequence
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1455 while (UTF8stride[*++p] != 0xFF) {}
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1456 }
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1457 return d;
0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1458 }
0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1459
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1460 void loadKeywords()
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1461 {
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1462 foreach(k; keywords)
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1463 idtable[k.str] = k;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1464 }
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1465
66 8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	1466 void error(MID id, ...)
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1467 {
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1468 char[][] arguments(TypeInfo[] tinfos, void* argptr)
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1469 {
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1470 char[][] args;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1471 foreach (ti; tinfos)
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1472 {
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1473 if (ti == typeid(char[]))
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1474 args ~= format(va_arg!(char[])(argptr));
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1475 else if (ti == typeid(int))
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1476 args ~= format(va_arg!(int)(argptr));
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1477 else if (ti == typeid(dchar))
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1478 args ~= format(va_arg!(dchar)(argptr));
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1479 else
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1480 assert(0, "argument type not supported yet.");
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1481 }
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1482 return args;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1483 }
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1484
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1485 errors ~= new Problem(Problem.Type.Lexer, id, loc, arguments(_arguments, _argptr));
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1486 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1487
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	1488 public TOK nextToken()
4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	1489 {
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	1490 scan(this.token);
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	1491 return this.token.type;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	1492 }
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	1493
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	1494 Token[] getTokens()
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	1495 {
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	1496 Token[] tokens;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	1497 while (nextToken() != TOK.EOF)
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	1498 tokens ~= this.token;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	1499 tokens ~= this.token;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	1500 return tokens;
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	1501 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1502
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1503 private void encodeUTF8(inout char[] str, dchar d)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1504 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1505 char[6] b;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1506 assert(d > 0x7F, "check for ASCII char before calling encodeUTF8().");
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1507 if (d < 0x800)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1508 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1509 b[0] = 0xC0 \| (d >> 6);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1510 b[1] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1511 str ~= b[0..2];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1512 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1513 else if (d < 0x10000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1514 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1515 b[0] = 0xE0 \| (d >> 12);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1516 b[1] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1517 b[2] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1518 str ~= b[0..3];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1519 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1520 else if (d < 0x200000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1521 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1522 b[0] = 0xF0 \| (d >> 18);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1523 b[1] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1524 b[2] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1525 b[3] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1526 str ~= b[0..4];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1527 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1528 else if (d < 0x4000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1529 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1530 b[0] = 0xF8 \| (d >> 24);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1531 b[1] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1532 b[2] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1533 b[3] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1534 b[4] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1535 str ~= b[0..5];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1536 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1537 else if (d < 0x80000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1538 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1539 b[0] = 0xFC \| (d >> 30);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1540 b[1] = 0x80 \| ((d >> 24) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1541 b[2] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1542 b[3] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1543 b[4] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1544 b[5] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1545 str ~= b[0..6];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1546 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1547 else
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1548 error(MID.InvalidUnicodeCharacter);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1549 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	1550 }
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1551
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1552 unittest
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1553 {
40 9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	1554 string[] toks = [
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	1555 ">", ">=", ">>", ">>=", ">>>", ">>>=", "<", "<=", "<>",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	1556 "<>=", "<<", "<<=", "!", "!<", "!>", "!<=", "!>=", "!<>",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	1557 "!<>=", ".", "..", "...", "&", "&&", "&=", "+", "++",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	1558 "+=", "-", "--", "-=", "=", "==", "~", "~=", "*",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	1559 "*=", "/", "/=", "^", "^=", "%", "%=", "(", ")",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	1560 "[", "]", "{", "}", ":", ";", "?", ",", "$"
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	1561 ];
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1562
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1563 char[] src;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1564
40 9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	1565 foreach (op; toks)
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1566 src ~= op ~ " ";
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1567
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1568 auto lx = new Lexer(src, "");
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1569 auto tokens = lx.getTokens();
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1570
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1571 tokens = tokens[0..$-1]; // exclude TOK.EOF
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1572
40 9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	1573 assert(tokens.length == toks.length );
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1574
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	1575 foreach (i, t; tokens)
40 9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	1576 assert(t.span == toks[i], std.string.format("Lexed '%s' but expected '%s'", t.span, toks[i]));
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	1577 }
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1578
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1579 unittest
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1580 {
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1581 // Numbers unittest
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1582 // 0L 0ULi 0_L 0_UL 0_Fi 0_e2 0_F 0_i 0x0U 0x0p2
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1583 }
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1584
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1585 /// ASCII character properties table.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1586 static const int ptable[256] = [
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1587 0, 0, 0, 0, 0, 0, 0, 0, 0,32, 0,32,32, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1588 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1589 32, 0, 0x2200, 0, 0, 0, 0, 0x2700, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1590 7, 7, 7, 7, 7, 7, 7, 7, 6, 6, 0, 0, 0, 0, 0, 0x3f00,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1591 0,12,12,12,12,12,12, 8, 8, 8, 8, 8, 8, 8, 8, 8,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1592 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0, 0x5c00, 0, 0,16,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1593 0, 0x70c, 0x80c,12,12,12, 0xc0c, 8, 8, 8, 8, 8, 8, 8, 0xa08, 8,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1594 8, 8, 0xd08, 8, 0x908, 8, 0xb08, 8, 8, 8, 8, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1595 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1596 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1597 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1598 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1599 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1600 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1601 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1602 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1603 ];
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1604
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1605 enum CProperty
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1606 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1607 Octal = 1,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1608 Digit = 1<<1,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1609 Hex = 1<<2,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1610 Alpha = 1<<3,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1611 Underscore = 1<<4,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1612 Whitespace = 1<<5
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1613 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1614
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1615 const uint EVMask = 0xFF00; // Bit mask for escape value
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1616
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1617 private alias CProperty CP;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1618 int isoctal(char c) { return ptable[c] & CP.Octal; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1619 int isdigit(char c) { return ptable[c] & CP.Digit; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1620 int ishexad(char c) { return ptable[c] & CP.Hex; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1621 int isalpha(char c) { return ptable[c] & CP.Alpha; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1622 int isalnum(char c) { return ptable[c] & (CP.Alpha \| CP.Digit); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1623 int isidbeg(char c) { return ptable[c] & (CP.Alpha \| CP.Underscore); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1624 int isident(char c) { return ptable[c] & (CP.Alpha \| CP.Underscore \| CP.Digit); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1625 int isspace(char c) { return ptable[c] & CP.Whitespace; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1626 int char2ev(char c) { return ptable[c] >> 8; /(ptable[c] & EVMask) >> 8;/ }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1627
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1628 version(gen_ptable)
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1629 static this()
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1630 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1631 alias ptable p;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1632 // Initialize character properties table.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1633 for (int i; i < p.length; ++i)
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1634 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1635 p[i] = 0;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1636 if ('0' <= i && i <= '7')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1637 p[i] \|= CP.Octal;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1638 if ('0' <= i && i <= '9')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1639 p[i] \|= CP.Digit;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1640 if (isdigit(i) \|\| 'a' <= i && i <= 'f' \|\| 'A' <= i && i <= 'F')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1641 p[i] \|= CP.Hex;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1642 if ('a' <= i && i <= 'z' \|\| 'A' <= i && i <= 'Z')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1643 p[i] \|= CP.Alpha;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1644 if (i == '_')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1645 p[i] \|= CP.Underscore;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1646 if (i == ' ' \|\| i == '\t' \|\| i == '\v' \|\| i == '\f')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1647 p[i] \|= CP.Whitespace;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1648 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1649 // Store escape sequence values in second byte.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1650 assert(CProperty.max <= ubyte.max, "character property flags and escape value byte overlap.");
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1651 p['\''] \|= 39 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1652 p['"'] \|= 34 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1653 p['?'] \|= 63 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1654 p['\\'] \|= 92 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1655 p['a'] \|= 7 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1656 p['b'] \|= 8 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1657 p['f'] \|= 12 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1658 p['n'] \|= 10 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1659 p['r'] \|= 13 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1660 p['t'] \|= 9 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1661 p['v'] \|= 11 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1662 // Print a formatted array literal.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1663 char[] array = "[\n";
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1664 for (int i; i < p.length; ++i)
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1665 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1666 int c = p[i];
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1667 array ~= std.string.format(c>255?" 0x%x,":"%2d,", c, ((i+1) % 16) ? "":"\n");
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1668 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1669 array[$-2..$] = "\n]";
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1670 writefln(array);
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	1671 }

Mercurial > projects > dil

annotate trunk/src/Lexer.d @ 67:996065105910