projects/dil: trunk/src/dil/Lexer.d annotate

annotate trunk/src/dil/Lexer.d @ 389:c4bfceab7246

Applied fixes and improvements to hex float scanner. '0x1234i' and '0x1234Li' were wrongly matched as hex floats. Refactored scanHexReal(). It's faster now and easier to read. Renamed MID.HexFloatMissingExpDigits to MID.HexFloatExpMustStartWithDigit. Renamed MID.FloatExponentDigitExpected to MID.FloatExpMustStartWidhtDigit. Fix in scanSpecialTokenSequence(): --p must come after if statement.

author	Aziz K?ksal <aziz.koeksal@gmail.com>
date	Wed, 12 Sep 2007 18:18:29 +0200
parents	ae154eceba65
children	4d36eea1bbc9

rev	line source
0 8ba2570de175 Initial import. aziz parents: diff changeset	1 /++
8ba2570de175 Initial import. aziz parents: diff changeset	2 Author: Aziz Köksal
249 32d354584b28 - Upgraded license notices to GPL3. aziz parents: 239 diff changeset	3 License: GPL3
0 8ba2570de175 Initial import. aziz parents: diff changeset	4 +/
326 4a7359b88c11 - Added package 'dil' to module declarations. aziz parents: 325 diff changeset	5 module dil.Lexer;
327 a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	6 import dil.Token;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	7 import dil.Information;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	8 import dil.Keywords;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	9 import dil.Identifier;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	10 import dil.Messages;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	11 import dil.HtmlEntities;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	12 import dil.Settings;
2 81c6cc33f5c8 - Initializing ptable with a precomputed array literal. aziz parents: 1 diff changeset	13 import std.stdio;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	14 import std.utf;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	15 import std.uni;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	16 import std.c.stdlib : strtof, strtod, strtold, getErrno, ERANGE;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	17 import std.c.time : time_t, time, ctime;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	18 import std.c.string : strlen;
66 8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	19 import std.string;
0 8ba2570de175 Initial import. aziz parents: diff changeset	20
11 dffcdaa7c47a - Added Unicode line and paragraph separators. aziz parents: 10 diff changeset	21 const char[3] LS = \u2028;
dffcdaa7c47a - Added Unicode line and paragraph separators. aziz parents: 10 diff changeset	22 const char[3] PS = \u2029;
dffcdaa7c47a - Added Unicode line and paragraph separators. aziz parents: 10 diff changeset	23
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	24 const dchar LSd = 0x2028;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	25 const dchar PSd = 0x2029;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	26
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	27 const uint _Z_ = 26; /// Control+Z
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	28
0 8ba2570de175 Initial import. aziz parents: diff changeset	29 class Lexer
8ba2570de175 Initial import. aziz parents: diff changeset	30 {
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	31 Token* head; /// The head of the doubly linked token list.
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	32 Token* tail; /// The tail of the linked list. Set in scan().
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	33 Token* token; /// Points to the current token in the token list.
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	34 string text;
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	35 char* p; /// Points to the current character in the source text.
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	36 char* end; /// Points one character past the end of the source text.
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	37
371 01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	38 uint loc = 1; /// Actual line of code.
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	39
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	40 uint loc_old; /// Store actual line number when #line token is parsed.
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	41 uint loc_hline; /// Line number set by #line.
17 9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	42
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	43 char[] fileName;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	44
69 24db7c5522d5 - Added module Information for compiler messages like warnings, info and errors to the user. aziz parents: 68 diff changeset	45 Information[] errors;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	46
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	47 // bool reportErrors;
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	48
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	49 Identifier[string] idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	50
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	51 this(string text, string fileName)
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	52 {
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	53 this.fileName = fileName;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	54
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	55 this.text = text;
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	56 if (text[$-1] != 0)
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	57 {
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	58 this.text.length = this.text.length + 1;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	59 this.text[$-1] = 0;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	60 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	61
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	62 this.p = this.text.ptr;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	63 this.end = this.p + this.text.length;
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	64 // this.reportErrors = true;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	65 loadKeywords();
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	66
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	67 this.head = new Token;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	68 this.head.type = TOK.HEAD;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	69 this.token = this.head;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	70 scanShebang();
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	71 }
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	72
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	73 ~this()
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	74 {
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	75 auto token = head.next;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	76 do
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	77 {
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	78 assert(token.type == TOK.EOF ? token == tail && token.next is null : 1);
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	79 delete token.prev;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	80 token = token.next;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	81 } while (token !is null)
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	82 delete tail;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	83 }
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	84
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	85 void scanShebang()
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	86 {
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	87 if (*p == '#' && p[1] == '!')
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	88 {
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	89 Token* t = new Token;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	90 t.start = p;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	91 t.type = TOK.Shebang;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	92 ++p;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	93 while (1)
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	94 {
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	95 t.end = p;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	96 switch (*++p)
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	97 {
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	98 case '\r':
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	99 if (p[1] == '\n')
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	100 ++p;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	101 case '\n':
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	102 ++p;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	103 ++loc;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	104 break;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	105 case 0, _Z_:
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	106 break;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	107 default:
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	108 if (*p & 128)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	109 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	110 auto c = decodeUTF8();
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	111 if (c == LSd \|\| c == PSd)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	112 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	113 }
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	114 continue;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	115 }
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	116 break; // Exit loop.
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	117 }
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	118 this.head.next = t;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	119 t.prev = this.head;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	120 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	121 }
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	122
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	123 void finalizeSpecialToken(ref Token t)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	124 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	125 assert(t.srcText[0..2] == "__");
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	126 switch (t.type)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	127 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	128 case TOK.FILE:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	129 t.str = this.fileName;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	130 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	131 case TOK.LINE:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	132 t.uint_ = this.loc;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	133 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	134 case TOK.DATE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	135 TOK.TIME,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	136 TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	137 time_t time_val;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	138 time(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	139 char* str = ctime(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	140 char[] time_str = str[0 .. strlen(str)];
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	141 switch (t.type)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	142 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	143 case TOK.DATE:
346 ce9e2e77743f - Fix: include one space when slicing time_str for __DATE__. aziz parents: 344 diff changeset	144 time_str = time_str[4..11] ~ time_str[20..24] ~ \0; break;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	145 case TOK.TIME:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	146 time_str = time_str[11..19] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	147 case TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	148 time_str = time_str[0..24] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	149 default: assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	150 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	151 t.str = time_str;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	152 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	153 case TOK.VENDOR:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	154 t.str = VENDOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	155 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	156 case TOK.VERSION:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	157 t.uint_ = VERSION_MAJOR*1000 + VERSION_MINOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	158 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	159 default:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	160 assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	161 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	162 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	163
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	164 public void scan(out Token t)
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	165 in
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	166 {
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	167 assert(text.ptr <= p && p < end);
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	168 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	169 out
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	170 {
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	171 assert(text.ptr <= t.start && t.start < end);
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	172 assert(text.ptr < t.end && t.end <= end, std.string.format(t.type));
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	173 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	174 body
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	175 {
10 3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	176 uint c = *p;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	177
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	178 while (1)
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	179 {
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	180 t.start = p;
16 476e8e55c1d4 - Added Whitespace to the character properties table. aziz parents: 15 diff changeset	181
212 c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	182 if (c == 0 \|\| c == _Z_)
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	183 {
212 c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	184 assert(p == 0 \|\| p == _Z_);
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	185 t.type = TOK.EOF;
17 9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	186 t.end = p;
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	187 tail = &t;
212 c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	188 assert(t.start == t.end);
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	189 return;
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	190 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	191
17 9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	192 if (c == '\n')
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	193 {
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	194 c = *++p;
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	195 ++loc;
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	196 continue;
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	197 }
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	198 else if (c == '\r')
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	199 {
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	200 c = *++p;
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	201 if (c != '\n')
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	202 ++loc;
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	203 continue;
9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	204 }
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	205 else if (c == LS[0] && p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2]))
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	206 {
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	207 p += 3;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	208 c = *p;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	209 continue;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	210 }
17 9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	211
13 e5211758b63c - Added isidbeg() function. aziz parents: 12 diff changeset	212 if (isidbeg(c))
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	213 {
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	214 if (c == 'r' && p[1] == '"' && ++p)
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	215 return scanRawStringLiteral(t);
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	216 if (c == 'x' && p[1] == '"')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	217 return scanHexStringLiteral(t);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	218 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	219 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	220 if (c == 'q' && p[1] == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	221 return scanDelimitedStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	222 if (c == 'q' && p[1] == '{')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	223 return scanTokenStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	224 }
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	225 Lidentifier:
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	226 do
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	227 { c = *++p; }
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	228 while (isident(c) \|\| c & 128 && isUniAlpha(decodeUTF8()))
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	229
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	230 t.end = p;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	231
65 6c21ae79fbb3 - Renamed function Token.span to Token.srcText. aziz parents: 64 diff changeset	232 string str = t.srcText;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	233 Identifier* id = str in idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	234
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	235 if (!id)
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	236 {
327 a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	237 idtable[str] = Identifier(TOK.Identifier, str);
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	238 id = str in idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	239 }
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	240 assert(id);
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	241 t.type = id.type;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	242 if (t.isSpecialToken)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	243 finalizeSpecialToken(t);
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	244 return;
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	245 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	246
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	247 if (isdigit(c))
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	248 return scanNumber(t);
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	249
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	250 if (c == '/')
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	251 {
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	252 c = *++p;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	253 switch(c)
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	254 {
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	255 case '=':
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	256 ++p;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	257 t.type = TOK.DivAssign;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	258 t.end = p;
cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	259 return;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	260 case '+':
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	261 uint level = 1;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	262 while (1)
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	263 {
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	264 c = *++p;
42 c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	265 LswitchNC: // only jumped to from default case of next switch(c)
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	266 switch (c)
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	267 {
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	268 case '\r':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	269 if (p[1] == '\n')
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	270 ++p;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	271 case '\n':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	272 ++loc;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	273 continue;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	274 case 0, _Z_:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	275 error(MID.UnterminatedNestedComment);
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	276 goto LreturnNC;
42 c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	277 default:
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	278 }
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	279
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	280 c <<= 8;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	281 c \|= *++p;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	282 switch (c)
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	283 {
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	284 case 0x2F2B: // /+
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	285 ++level;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	286 continue;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	287 case 0x2B2F: // +/
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	288 if (--level == 0)
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	289 {
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	290 ++p;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	291 LreturnNC:
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	292 t.type = TOK.Comment;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	293 t.end = p;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	294 return;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	295 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	296 continue;
42 c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	297 case 0xE280: // LS[0..1] \|\| PS[0..1]
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	298 if (p[1] == LS[2] \|\| p[1] == PS[2])
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	299 {
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	300 ++loc;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	301 ++p;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	302 }
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	303 continue;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	304 default:
42 c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	305 c &= char.max;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	306 goto LswitchNC;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	307 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	308 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	309 case '*':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	310 while (1)
7 07e45c06a024 - Parsing nested comments correctly now. aziz parents: 5 diff changeset	311 {
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	312 c = *++p;
1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	313 LswitchBC: // only jumped to from default case of next switch(c)
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	314 switch (c)
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	315 {
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	316 case '\r':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	317 if (p[1] == '\n')
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	318 ++p;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	319 case '\n':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	320 ++loc;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	321 continue;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	322 case 0, _Z_:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	323 error(MID.UnterminatedBlockComment);
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	324 goto LreturnBC;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	325 default:
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	326 }
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	327
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	328 c <<= 8;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	329 c \|= *++p;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	330 switch (c)
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	331 {
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	332 case 0x2A2F: // */
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	333 ++p;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	334 LreturnBC:
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	335 t.type = TOK.Comment;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	336 t.end = p;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	337 return;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	338 case 0xE280: // LS[0..1] \|\| PS[0..1]
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	339 if (p[1] == LS[2] \|\| p[1] == PS[2])
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	340 {
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	341 ++loc;
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	342 ++p;
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	343 }
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	344 continue;
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	345 default:
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	346 c &= char.max;
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	347 goto LswitchBC;
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	348 }
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	349 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	350 assert(0);
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	351 case '/':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	352 while (1)
7 07e45c06a024 - Parsing nested comments correctly now. aziz parents: 5 diff changeset	353 {
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	354 c = *++p;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	355 switch (c)
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	356 {
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	357 case '\r':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	358 if (p[1] == '\n')
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	359 ++p;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	360 case '\n':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	361 case 0, _Z_:
11 dffcdaa7c47a - Added Unicode line and paragraph separators. aziz parents: 10 diff changeset	362 break;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	363 case LS[0]:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	364 if (p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2]))
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	365 break;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	366 continue;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	367 default:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	368 continue;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	369 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	370 t.type = TOK.Comment;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	371 t.end = p;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	372 return;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	373 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	374 default:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	375 t.type = TOK.Div;
10 3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	376 t.end = p;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	377 return;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	378 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	379 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	380
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	381 switch (c)
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	382 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	383 case '\'':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	384 return scanCharacterLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	385 case '`':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	386 return scanRawStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	387 case '"':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	388 return scanNormalStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	389 case '\\':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	390 char[] buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	391 do
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	392 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	393 c = scanEscapeSequence();
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	394 if (c < 128)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	395 buffer ~= c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	396 else
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	397 encodeUTF8(buffer, c);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	398 } while (*p == '\\')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	399 buffer ~= 0;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	400 t.type = TOK.String;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	401 t.str = buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	402 t.end = p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	403 return;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	404 case '>': /* > >= >> >>= >>> >>>= */
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	405 c = *++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	406 switch (c)
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	407 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	408 case '=':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	409 t.type = TOK.GreaterEqual;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	410 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	411 case '>':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	412 if (p[1] == '>')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	413 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	414 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	415 if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	416 { ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	417 t.type = TOK.URShiftAssign;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	418 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	419 else
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	420 t.type = TOK.URShift;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	421 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	422 else if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	423 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	424 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	425 t.type = TOK.RShiftAssign;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	426 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	427 else
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	428 t.type = TOK.RShift;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	429 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	430 default:
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	431 t.type = TOK.Greater;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	432 goto Lcommon2;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	433 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	434 assert(0);
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	435 case '<': /* < <= <> <>= << <<= */
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	436 c = *++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	437 switch (c)
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	438 {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	439 case '=':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	440 t.type = TOK.LessEqual;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	441 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	442 case '<':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	443 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	444 ++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	445 t.type = TOK.LShiftAssign;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	446 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	447 else
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	448 t.type = TOK.LShift;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	449 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	450 case '>':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	451 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	452 ++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	453 t.type = TOK.LorEorG;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	454 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	455 else
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	456 t.type = TOK.LorG;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	457 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	458 default:
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	459 t.type = TOK.Less;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	460 goto Lcommon2;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	461 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	462 assert(0);
37 7f3bcb97d017 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 36 diff changeset	463 case '!': /* ! !< !> !<= !>= !<> !<>= */
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	464 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	465 switch (c)
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	466 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	467 case '<':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	468 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	469 if (c == '>')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	470 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	471 if (p[1] == '=') {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	472 ++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	473 t.type = TOK.Unordered;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	474 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	475 else
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	476 t.type = TOK.UorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	477 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	478 else if (c == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	479 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	480 t.type = TOK.UorG;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	481 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	482 else {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	483 t.type = TOK.UorGorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	484 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	485 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	486 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	487 case '>':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	488 if (p[1] == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	489 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	490 ++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	491 t.type = TOK.UorL;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	492 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	493 else
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	494 t.type = TOK.UorLorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	495 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	496 case '=':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	497 t.type = TOK.NotEqual;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	498 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	499 default:
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	500 t.type = TOK.Not;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	501 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	502 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	503 assert(0);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	504 case '.': /* . .[0-9] .. ... */
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	505 if (p[1] == '.')
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	506 {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	507 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	508 if (p[1] == '.') {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	509 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	510 t.type = TOK.Ellipses;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	511 }
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	512 else
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	513 t.type = TOK.Slice;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	514 }
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	515 else if (isdigit(p[1]))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	516 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	517 return scanReal(t);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	518 }
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	519 else
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	520 t.type = TOK.Dot;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	521 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	522 case '\|': /* \| \|\| \|= */
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	523 c = *++p;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	524 if (c == '=')
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	525 t.type = TOK.OrAssign;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	526 else if (c == '\|')
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	527 t.type = TOK.OrLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	528 else {
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	529 t.type = TOK.OrBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	530 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	531 }
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	532 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	533 case '&': /* & && &= */
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	534 c = *++p;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	535 if (c == '=')
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	536 t.type = TOK.AndAssign;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	537 else if (c == '&')
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	538 t.type = TOK.AndLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	539 else {
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	540 t.type = TOK.AndBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	541 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	542 }
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	543 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	544 case '+': /* + ++ += */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	545 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	546 if (c == '=')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	547 t.type = TOK.PlusAssign;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	548 else if (c == '+')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	549 t.type = TOK.PlusPlus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	550 else {
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	551 t.type = TOK.Plus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	552 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	553 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	554 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	555 case '-': /* - -- -= */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	556 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	557 if (c == '=')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	558 t.type = TOK.MinusAssign;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	559 else if (c == '-')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	560 t.type = TOK.MinusMinus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	561 else {
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	562 t.type = TOK.Minus;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	563 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	564 }
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	565 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	566 case '=': /* = == */
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	567 if (p[1] == '=') {
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	568 ++p;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	569 t.type = TOK.Equal;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	570 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	571 else
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	572 t.type = TOK.Assign;
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	573 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	574 case '~': /* ~ ~= */
27 43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	575 if (p[1] == '=') {
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	576 ++p;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	577 t.type = TOK.CatAssign;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	578 }
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	579 else
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	580 t.type = TOK.Tilde;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	581 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	582 case '': / * = /
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	583 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	584 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	585 t.type = TOK.MulAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	586 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	587 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	588 t.type = TOK.Mul;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	589 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	590 case '^': /* ^ ^= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	591 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	592 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	593 t.type = TOK.XorAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	594 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	595 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	596 t.type = TOK.Xor;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	597 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	598 case '%': /* % %= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	599 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	600 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	601 t.type = TOK.ModAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	602 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	603 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	604 t.type = TOK.Mod;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	605 goto Lcommon;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	606 // Single character tokens:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	607 case '(':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	608 t.type = TOK.LParen;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	609 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	610 case ')':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	611 t.type = TOK.RParen;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	612 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	613 case '[':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	614 t.type = TOK.LBracket;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	615 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	616 case ']':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	617 t.type = TOK.RBracket;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	618 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	619 case '{':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	620 t.type = TOK.LBrace;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	621 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	622 case '}':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	623 t.type = TOK.RBrace;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	624 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	625 case ':':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	626 t.type = TOK.Colon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	627 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	628 case ';':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	629 t.type = TOK.Semicolon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	630 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	631 case '?':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	632 t.type = TOK.Question;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	633 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	634 case ',':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	635 t.type = TOK.Comma;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	636 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	637 case '$':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	638 t.type = TOK.Dollar;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	639 Lcommon:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	640 ++p;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	641 Lcommon2:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	642 t.end = p;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	643 return;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	644 case '#':
360 b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence(). aziz parents: 350 diff changeset	645 return scanSpecialTokenSequence(t);
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	646 default:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	647 }
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	648
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	649 if (c & 128 && isUniAlpha(decodeUTF8()))
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	650 goto Lidentifier;
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	651 c = *++p;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	652 }
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	653 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	654
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	655 void scanNormalStringLiteral(ref Token t)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	656 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	657 assert(*p == '"');
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	658 char[] buffer;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	659 t.type = TOK.String;
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	660 uint c;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	661 while (1)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	662 {
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	663 c = *++p;
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	664 switch (c)
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	665 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	666 case '"':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	667 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	668 Lreturn:
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	669 buffer ~= 0;
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	670 t.str = buffer;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	671 t.pf = scanPostfix();
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	672 t.end = p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	673 return;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	674 case '\\':
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	675 c = scanEscapeSequence();
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	676 --p;
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	677 if (c & 128)
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	678 encodeUTF8(buffer, c);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	679 else
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	680 break;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	681 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	682 case '\r':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	683 if (p[1] == '\n')
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	684 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	685 case '\n':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	686 ++loc;
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	687 c = '\n'; // Convert EndOfLine to \n.
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	688 break;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	689 case 0, _Z_:
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	690 error(MID.UnterminatedString);
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	691 goto Lreturn;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	692 default:
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	693 if (c & 128)
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	694 {
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	695 // char* begin = p;
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	696 c = decodeUTF8();
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	697 if (c == LSd \|\| c == PSd)
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	698 goto case '\n';
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	699
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	700 // We don't copy per pointer because we might include
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	701 // invalid, skipped utf-8 sequences. See decodeUTF8().
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	702 // ++p;
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	703 // buffer ~= begin[0 .. p - begin];
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	704 encodeUTF8(buffer, c);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	705 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	706 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	707 }
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	708 // Copy ASCII character.
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	709 buffer ~= c;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	710 }
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	711 assert(0);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	712 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	713
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	714 void scanCharacterLiteral(ref Token t)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	715 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	716 assert(*p == '\'');
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	717 MID id = MID.UnterminatedCharacterLiteral;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	718 ++p;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	719 TOK type = TOK.CharLiteral;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	720 switch (*p)
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	721 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	722 case '\\':
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	723 switch (p[1])
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	724 {
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	725 case 'u':
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	726 type = TOK.WCharLiteral; break;
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	727 case 'U':
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	728 type = TOK.DCharLiteral; break;
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	729 default:
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	730 }
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	731 t.dchar_ = scanEscapeSequence();
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	732 break;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	733 case '\'':
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	734 ++p;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	735 id = MID.EmptyCharacterLiteral;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	736 case '\n', '\r', 0, _Z_:
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	737 goto Lerr;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	738 default:
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	739 uint c = *p;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	740 if (c & 128)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	741 {
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	742 c = decodeUTF8();
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	743 if (c == LSd \|\| c == PSd)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	744 goto Lerr;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	745 if (c <= 0xFFFF)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	746 type = TOK.WCharLiteral;
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	747 else
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	748 type = TOK.DCharLiteral;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	749 }
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	750 t.dchar_ = c;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	751 ++p;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	752 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	753
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	754 if (*p == '\'')
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	755 ++p;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	756 else
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	757 Lerr:
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	758 error(id);
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	759 t.type = type;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	760 t.end = p;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	761 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	762
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	763 char scanPostfix()
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	764 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	765 switch (*p)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	766 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	767 case 'c':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	768 case 'w':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	769 case 'd':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	770 return *p++;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	771 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	772 return 0;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	773 }
112 004d98df65af - Implemented parseInterfaceDeclaration(). aziz parents: 105 diff changeset	774 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	775 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	776
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	777 void scanRawStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	778 {
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	779 uint delim = *p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	780 assert(delim == '`' \|\| delim == '"' && p[-1] == 'r');
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	781 t.type = TOK.String;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	782 char[] buffer;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	783 uint c;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	784 while (1)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	785 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	786 c = *++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	787 switch (c)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	788 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	789 case '\r':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	790 if (p[1] == '\n')
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	791 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	792 case '\n':
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	793 c = '\n'; // Convert EndOfLine ('\r','\r\n','\n',LS,PS) to '\n'
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	794 ++loc;
52 f65a83c27638 - Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well. aziz parents: 51 diff changeset	795 break;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	796 case '`':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	797 case '"':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	798 if (c == delim)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	799 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	800 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	801 t.pf = scanPostfix();
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	802 Lreturn:
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	803 t.str = buffer ~ '\0';
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	804 t.end = p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	805 return;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	806 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	807 break;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	808 case 0, _Z_:
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	809 if (delim == 'r')
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	810 error(MID.UnterminatedRawString);
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	811 else
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	812 error(MID.UnterminatedBackQuoteString);
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	813 goto Lreturn;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	814 default:
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	815 if (c & 128)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	816 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	817 c = decodeUTF8();
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	818 if (c == LSd \|\| c == PSd)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	819 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	820 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	821 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	822 }
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	823 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	824 buffer ~= c; // copy character to buffer
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	825 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	826 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	827 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	828
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	829 void scanHexStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	830 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	831 assert(p[0] == 'x' && p[1] == '"');
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	832 t.type = TOK.String;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	833
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	834 uint c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	835 ubyte[] buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	836 ubyte h; // hex number
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	837 uint n; // number of hex digits
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	838
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	839 ++p;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	840 while (1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	841 {
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	842 c = *++p;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	843 switch (c)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	844 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	845 case '"':
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	846 ++p;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	847 if (n & 1)
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	848 error(MID.OddNumberOfDigitsInHexString);
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	849 t.pf = scanPostfix();
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	850 Lreturn:
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	851 buffer ~= 0;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	852 t.str = cast(string) buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	853 t.end = p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	854 return;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	855 case '\r':
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	856 if (p[1] == '\n')
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	857 ++p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	858 case '\n':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	859 ++loc;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	860 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	861 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	862 if (ishexad(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	863 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	864 if (c <= '9')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	865 c -= '0';
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	866 else if (c <= 'F')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	867 c -= 'A' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	868 else
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	869 c -= 'a' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	870
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	871 if (n & 1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	872 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	873 h <<= 4;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	874 h \|= c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	875 buffer ~= h;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	876 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	877 else
112 004d98df65af - Implemented parseInterfaceDeclaration(). aziz parents: 105 diff changeset	878 h = cast(ubyte)c;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	879 ++n;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	880 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	881 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	882 else if (isspace(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	883 continue;
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	884
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	885 if (c & 128)
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	886 {
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	887 c = decodeUTF8();
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	888 if (c == LSd \|\| c == PSd)
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	889 {
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	890 ++p; ++p;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	891 ++loc;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	892 continue;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	893 }
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	894 }
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	895 else if (c == 0 \|\| c == _Z_)
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	896 {
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	897 error(MID.UnterminatedHexString);
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	898 t.pf = 0;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	899 goto Lreturn;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	900 }
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	901 error(MID.NonHexCharInHexString, cast(dchar)c);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	902 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	903 }
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	904 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	905 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	906
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	907 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	908 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	909 void scanDelimitedStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	910 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	911 assert(p[0] == 'q' && p[1] == '"');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	912 t.type = TOK.String;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	913
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	914 char[] buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	915 dchar opening_delim, // 0 if no nested delimiter or '[', '(', '<', '{'
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	916 closing_delim; // Will be ']', ')', '>', '}', any other character
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	917 // or the first, decoded character of an identifier.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	918 char[] str_delim; // Identifier delimiter
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	919 uint level = 1;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	920
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	921 ++p; ++p; // Skip q"
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	922 uint c = *p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	923 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	924 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	925 case '(':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	926 opening_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	927 closing_delim = ')'; // *p + 1
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	928 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	929 case '[', '<', '{':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	930 opening_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	931 closing_delim = c + 2; // Get to closing counterpart. Feature of ASCII table.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	932 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	933 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	934 char* begin = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	935 closing_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	936 // TODO: What to do about newlines? Skip or accept as delimiter?
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	937 // TODO: Check for non-printable characters?
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	938 if (c & 128)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	939 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	940 closing_delim = decodeUTF8();
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	941 if (!isUniAlpha(c))
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	942 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	943 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	944 else if (!isidbeg(c))
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	945 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	946 // Parse identifier + newline
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	947 do
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	948 { c = *++p; }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	949 while (isident(c) \|\| c & 128 && isUniAlpha(decodeUTF8()))
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	950 // Store identifier
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	951 str_delim = begin[0..p-begin];
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	952 // Scan newline
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	953 switch (*p)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	954 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	955 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	956 if (p[1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	957 ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	958 case '\n':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	959 ++loc;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	960 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	961 case LS[0]:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	962 if (p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2]))
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	963 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	964 ++p; ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	965 ++loc;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	966 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	967 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	968 // goto default;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	969 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	970 // TODO: error(MID.ExpectedNewlineAfterIdentDelim);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	971 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	972 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	973
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	974 bool checkStringDelim(char* p)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	975 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	976 assert(str_delim.length != 0);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	977 if (end-p >= str_delim.length && // Check remaining length.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	978 p[0..str_delim.length] == str_delim) // Compare.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	979 return true;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	980 return false;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	981 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	982
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	983 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	984 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	985 c = *++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	986 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	987 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	988 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	989 if (p[1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	990 ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	991 case '\n':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	992 c = '\n'; // Convert EndOfLine ('\r','\r\n','\n',LS,PS) to '\n'
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	993 ++loc;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	994 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	995 case 0, _Z_:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	996 // error(MID.UnterminatedDelimitedString);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	997 goto Lreturn3;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	998 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	999 if (c & 128)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1000 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1001 auto begin = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1002 c = decodeUTF8();
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1003 if (c == LSd \|\| c == PSd)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1004 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1005 if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1006 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1007 if (str_delim.length && checkStringDelim(begin))
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1008 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1009 p = begin + str_delim.length;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1010 goto Lreturn2;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1011 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1012 assert(level == 1);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1013 --level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1014 goto Lreturn;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1015 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1016 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1017 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1018 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1019 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1020 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1021 if (c == opening_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1022 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1023 else if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1024 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1025 if (str_delim.length && checkStringDelim(p))
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1026 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1027 p += str_delim.length;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1028 goto Lreturn2;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1029 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1030 if (--level == 0)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1031 goto Lreturn;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1032 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1033 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1034 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1035 buffer ~= c; // copy character to buffer
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1036 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1037 Lreturn:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1038 assert(*p == closing_delim);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1039 assert(level == 0);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1040 ++p; // Skip closing delimiter.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1041 Lreturn2:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1042 if (*p == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1043 ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1044 // else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1045 // TODO: error(MID.ExpectedDblQuoteAfterDelim, str_delim.length ? str_delim : p[-1]);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1046
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1047 t.pf = scanPostfix();
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1048 Lreturn3:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1049 t.str = buffer ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1050 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1051 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1052
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1053 void scanTokenStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1054 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1055 assert(p[0] == 'q' && p[1] == '{');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1056 t.type = TOK.String;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1057 // Copy members that might be changed by subsequent tokens. Like #line for example.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1058 auto loc_old = this.loc_old;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1059 auto loc_hline = this.loc_hline;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1060 auto filePath = this.fileName;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1061
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1062 uint loc = this.loc;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1063 uint level = 1;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1064
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1065 ++p; ++p; // Skip q{
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1066
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1067 auto prev_t = &t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1068 Token* token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1069 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1070 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1071 token = new Token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1072 scan(*token);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1073 // Save the tokens in a doubly linked list.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1074 // Could be useful for various tools.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1075 token.prev = prev_t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1076 prev_t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1077 prev_t = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1078 switch (token.type)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1079 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1080 case TOK.LBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1081 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1082 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1083 case TOK.RBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1084 if (--level == 0)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1085 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1086 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1087 t.next = null;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1088 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1089 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1090 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1091 case TOK.EOF:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1092 // TODO: error(MID.UnterminatedTokenString);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1093 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1094 t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1095 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1096 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1097 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1098 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1099 break; // Exit loop.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1100 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1101
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1102 assert(token.type == TOK.RBrace \|\| token.type == TOK.EOF);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1103 assert(token.type == TOK.RBrace && t.next is null \|\|
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1104 token.type == TOK.EOF && t.next !is null);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1105
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1106 char[] buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1107 // token points to } or EOF
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1108 if (token.type == TOK.EOF)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1109 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1110 t.end = token.start;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1111 buffer = t.srcText[2..$].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1112 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1113 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1114 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1115 // Assign to buffer before scanPostfix().
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1116 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1117 buffer = t.srcText[2..$-1].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1118 t.pf = scanPostfix();
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1119 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1120 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1121 // Convert EndOfLines to '\n'
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1122 if (loc != this.loc)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1123 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1124 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1125 uint i, j;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1126 for (; i < buffer.length; ++i)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1127 switch (buffer[i])
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1128 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1129 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1130 if (buffer[i+1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1131 ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1132 case '\n':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1133 buffer[j++] = '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1134 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1135 case LS[0]:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1136 auto b = buffer[i..$];
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1137 if (b[1] == LS[1] && (b[2] == LS[2] \|\| b[2] == PS[2]))
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1138 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1139 ++i; ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1140 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1141 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1142 // goto default;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1143 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1144 buffer[j++] = buffer[i]; // Copy character
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1145 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1146 buffer.length = j; // Adjust length
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1147 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1148 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1149 t.str = buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1150
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1151 // Restore possibly changed members.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1152 this.loc_old = loc_old;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1153 this.loc_hline = loc_hline;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1154 this.fileName = filePath;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1155 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1156 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1157
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1158 dchar scanEscapeSequence()
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1159 {
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1160 assert(*p == '\\');
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1161 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1162 uint c = char2ev(*p);
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1163 if (c)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1164 {
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1165 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1166 return c;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1167 }
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1168 uint digits = 2;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1169
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1170 switch (*p)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1171 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1172 case 'x':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1173 c = 0;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1174 while (1)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1175 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1176 ++p;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1177 if (ishexad(*p))
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1178 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1179 c *= 16;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1180 if (*p <= '9')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1181 c += *p - '0';
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1182 else if (*p <= 'F')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1183 c += *p - 'A' + 10;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1184 else
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1185 c += *p - 'a' + 10;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1186
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1187 if (!--digits)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1188 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1189 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1190 break;
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1191 }
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1192 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1193 else
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1194 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1195 error(MID.InsufficientHexDigits);
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1196 break;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1197 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1198 }
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1199 if (!isValidDchar(c))
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1200 error(MID.InvalidUnicodeCharacter);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1201 break;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1202 case 'u':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1203 digits = 4;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1204 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1205 case 'U':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1206 digits = 8;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1207 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1208 default:
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1209 if (isoctal(*p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1210 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1211 c = 0;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1212 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1213 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1214 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1215 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1216 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1217 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1218 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1219 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1220 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1221 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1222 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1223 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1224 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1225 else if(*p == '&')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1226 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1227 if (isalpha(*++p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1228 {
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1229 auto begin = p;
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1230 while (isalnum(*++p))
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1231 {}
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1232
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1233 if (*p == ';')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1234 {
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1235 c = entity2Unicode(begin[0..p - begin]);
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1236 ++p; // Skip ;
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1237 if (c == 0xFFFF)
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1238 error(MID.UndefinedHTMLEntity, (begin-1)[0..p-(begin-1)]);
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1239 }
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1240 else
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1241 error(MID.UnterminatedHTMLEntity);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1242 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1243 else
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1244 error(MID.InvalidBeginHTMLEntity);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1245 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1246 else
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1247 {
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1248 dchar d = *p;
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1249 char[] str = `\`;
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1250 if (d & 128)
387 ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1251 encodeUTF8(str, decodeUTF8());
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1252 else
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1253 str ~= d;
387 ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1254 ++p;
ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1255 // TODO: check for unprintable character?
ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1256 error(MID.UndefinedEscapeSequence, str);
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1257 }
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1258 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1259
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1260 return c;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1261 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1262
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1263 /*
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1264 IntegerLiteral:= (Dec\|Hex\|Bin\|Oct)Suffix?
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1265 Dec:= (0\|[1-9][0-9_]*)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1266 Hex:= 0[xX] HexDigits
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1267 Bin:= 0[bB][01_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1268 Oct:= 0[0-7_]+
68 7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	1269 Suffix:= (L[uU]?\|[uU]L?)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1270 HexDigits:= [0-9a-zA-Z_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1271
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1272 Invalid: "0b_", "0x_", "._"
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1273 */
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1274 void scanNumber(ref Token t)
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1275 {
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1276 ulong ulong_;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1277 bool overflow;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1278 bool isDecimal;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1279 size_t digits;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1280
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1281 if (*p != '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1282 goto LscanInteger;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1283 ++p; // skip zero
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1284 // check for xX bB ...
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1285 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1286 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1287 case 'x','X':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1288 goto LscanHex;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1289 case 'b','B':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1290 goto LscanBin;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1291 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1292 if (p[1] == 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1293 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1294 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1295 if (p[1] == '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1296 break;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1297 case 'i','f','F', 'e', 'E': // Imaginary and float literal suffix
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1298 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1299 default:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1300 if (p == '_' \|\| isoctal(p))
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1301 goto LscanOct;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1302 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1303
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1304 // Number 0
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1305 assert(p[-1] == '0');
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1306 assert(ulong_ == 0);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1307 isDecimal = true;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1308 goto Lfinalize;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1309
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1310 LscanInteger:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1311 assert(p != 0 && isdigit(p));
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1312 isDecimal = true;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1313 goto Lenter_loop_int;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1314 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1315 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1316 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1317 continue;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1318 if (!isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1319 break;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1320 Lenter_loop_int:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1321 if (ulong_ < ulong.max/10 \|\| (ulong_ == ulong.max/10 && *p <= '5'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1322 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1323 ulong_ *= 10;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1324 ulong_ += *p - '0';
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1325 continue;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1326 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1327 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1328 overflow = true;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1329 while (isdigit(*++p)) {}
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1330 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1331 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1332
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1333 // The number could be a float, so check overflow below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1334 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1335 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1336 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1337 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1338 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1339 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1340 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1341 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1342 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1343 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1344 goto LscanReal;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1345 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1346 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1347
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1348 if (overflow)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1349 error(MID.OverflowDecimalNumber);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1350
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1351 assert((isdigit(p[-1]) \|\| p[-1] == '_') && !isdigit(p) && p != '_');
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1352 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1353
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1354 LscanHex:
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1355 assert(digits == 0);
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1356 assert(*p == 'x');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1357 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1358 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1359 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1360 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1361 if (!ishexad(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1362 break;
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1363 ++digits;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1364 ulong_ *= 16;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1365 if (*p <= '9')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1366 ulong_ += *p - '0';
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1367 else if (*p <= 'F')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1368 ulong_ += *p - 'A' + 10;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1369 else
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1370 ulong_ += *p - 'a' + 10;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1371 }
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1372
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1373 assert(ishexad(p[-1]) \|\| p[-1] == '_' \|\| p[-1] == 'x');
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1374 assert(!ishexad(p) && p != '_');
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1375
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1376 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1377 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1378 case '.':
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1379 if (p[1] == '.')
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1380 break;
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1381 case 'p', 'P':
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1382 return scanHexReal(t);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1383 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1384 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1385
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1386 if (digits == 0)
59 3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	1387 error(MID.NoDigitsInHexNumber);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1388 else if (digits > 16)
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1389 error(MID.OverflowHexNumber);
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1390
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1391 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1392
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1393 LscanBin:
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1394 assert(digits == 0);
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1395 assert(*p == 'b');
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1396 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1397 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1398 if (*++p == '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1399 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1400 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1401 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1402 }
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1403 else if (*p == '1')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1404 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1405 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1406 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1407 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1408 }
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1409 else if (*p == '_')
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1410 continue;
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1411 else
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1412 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1413 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1414
59 3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	1415 if (digits == 0)
3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	1416 error(MID.NoDigitsInBinNumber);
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1417 else if (digits > 64)
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1418 error(MID.OverflowBinaryNumber);
59 3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	1419
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1420 assert(p[-1] == '0' \|\| p[-1] == '1' \|\| p[-1] == '_', p[-1] ~ "");
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1421 assert( !(p == '0' \|\| p == '1' \|\| *p == '_') );
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1422 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1423
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1424 LscanOct:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1425 assert(p == '_' \|\| isoctal(p));
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1426 if (*p != '_')
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1427 goto Lenter_loop_oct;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1428 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1429 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1430 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1431 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1432 if (!isoctal(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1433 break;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1434 Lenter_loop_oct:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1435 if (ulong_ < ulong.max/2 \|\| (ulong_ == ulong.max/2 && *p <= '1'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1436 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1437 ulong_ *= 8;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1438 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1439 ++p;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1440 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1441 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1442 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1443 overflow = true;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1444 while (isdigit(*++p)) {}
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1445 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1446 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1447
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1448 bool hasDecimalDigits;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1449 if (isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1450 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1451 hasDecimalDigits = true;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1452 while (isdigit(*++p)) {}
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1453 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1454
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1455 // The number could be a float, so check errors below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1456 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1457 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1458 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1459 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1460 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1461 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1462 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1463 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1464 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1465 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1466 goto LscanReal;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1467 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1468 }
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1469
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1470 if (hasDecimalDigits)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1471 error(MID.OctalNumberHasDecimals);
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1472 if (overflow)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1473 error(MID.OverflowOctalNumber);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1474 // goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1475
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1476 Lfinalize:
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1477 enum Suffix
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1478 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1479 None = 0,
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1480 Unsigned = 1,
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1481 Long = 2
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1482 }
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1483
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1484 Suffix suffix;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1485 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1486 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1487 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1488 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1489 case 'L':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1490 if (suffix & Suffix.Long)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1491 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1492 suffix \|= Suffix.Long;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1493 ++p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1494 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1495 case 'u', 'U':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1496 if (suffix & Suffix.Unsigned)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1497 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1498 suffix \|= Suffix.Unsigned;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1499 ++p;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1500 continue;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1501 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1502 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1503 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1504 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1505 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1506
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1507 switch (suffix)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1508 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1509 case Suffix.None:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1510 if (ulong_ & 0x8000000000000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1511 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1512 if (isDecimal)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1513 error(MID.OverflowDecimalSign);
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1514 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1515 }
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1516 else if (ulong_ & 0xFFFFFFFF00000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1517 t.type = TOK.Int64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1518 else if (ulong_ & 0x80000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1519 t.type = isDecimal ? TOK.Int64 : TOK.Uint32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1520 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1521 t.type = TOK.Int32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1522 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1523 case Suffix.Unsigned:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1524 if (ulong_ & 0xFFFFFFFF00000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1525 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1526 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1527 t.type = TOK.Uint32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1528 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1529 case Suffix.Long:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1530 if (ulong_ & 0x8000000000000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1531 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1532 if (isDecimal)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1533 error(MID.OverflowDecimalSign);
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1534 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1535 }
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1536 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1537 t.type = TOK.Int64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1538 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1539 case Suffix.Unsigned \| Suffix.Long:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1540 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1541 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1542 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1543 assert(0);
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1544 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1545 t.ulong_ = ulong_;
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1546 t.end = p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1547 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1548 LscanReal:
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1549 scanReal(t);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1550 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1551 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1552
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1553 /*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1554 FloatLiteral:= Float[fFL]?i?
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1555 Float:= DecFloat \| HexFloat
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1556 DecFloat:= ([0-9][0-9_][.][0-9_]DecExponent?) \| [.][0-9][0-9_]DecExponent? \| [0-9][0-9_]DecExponent
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1557 DecExponent:= [eE][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1558 HexFloat:= 0[xX](HexDigits[.]HexDigits \| [.][0-9a-zA-Z]HexDigits? \| HexDigits)HexExponent
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1559 HexExponent:= [pP][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1560 */
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1561 void scanReal(ref Token t)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1562 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1563 if (*p == '.')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1564 // This function was called by scan() or scanNumber().
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1565 while (isdigit(++p) \|\| p == '_') {}
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1566 else
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1567 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1568 // This function was called by scanNumber().
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1569 debug switch (*p)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1570 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1571 case 'L':
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1572 if (p[1] != 'i')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1573 assert(0);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1574 case 'i', 'f', 'F', 'e', 'E': break;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1575 default: assert(0);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1576 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1577 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1578
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1579 // Scan exponent.
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1580 if (p == 'e' \|\| p == 'E')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1581 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1582 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1583 if (p == '-' \|\| p == '+')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1584 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1585 if (!isdigit(*p))
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1586 error(MID.FloatExpMustStartWithDigit);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1587 else
64 dd4c5a0e47dd - Improved while loop. aziz parents: 63 diff changeset	1588 while (isdigit(++p) \|\| p == '_') {}
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1589 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1590
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1591 // Copy string to buffer ignoring underscores.
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1592 char[] buffer;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1593 char* end = p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1594 p = t.start;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1595 do
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1596 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1597 if (*p == '_')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1598 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1599 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1600 continue;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1601 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1602 buffer ~= *p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1603 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1604 } while (p != end)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1605 buffer ~= 0;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1606 finalizeFloat(t, buffer);
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1607 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1608
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1609 void scanHexReal(ref Token t)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1610 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1611 assert(p == '.' \|\| p == 'p' \|\| *p == 'P');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1612 MID mid;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1613 if (*p == '.')
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1614 while (ishexad(++p) \|\| p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1615 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1616 // Decimal exponent is required.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1617 if (p != 'p' && p != 'P')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1618 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1619 mid = MID.HexFloatExponentRequired;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1620 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1621 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1622 // Scan exponent
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1623 assert(p == 'p' \|\| p == 'P');
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1624 if (!isdigit(*++p))
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1625 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1626 mid = MID.HexFloatExpMustStartWithDigit;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1627 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1628 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1629 while (isdigit(++p) \|\| p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1630 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1631 // Copy whole number and remove underscores from buffer.
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1632 char[] buffer = t.start[0..p-t.start].dup;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1633 uint j;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1634 foreach (c; buffer)
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1635 if (c != '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1636 buffer[j++] = c;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1637 buffer.length = j; // Adjust length.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1638 buffer ~= 0; // Terminate for C functions.
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1639 finalizeFloat(t, buffer);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1640 return;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1641 Lerr:
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1642 t.type = TOK.Float32;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1643 t.end = p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1644 error(mid);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1645 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1646
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1647 void finalizeFloat(ref Token t, string buffer)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1648 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1649 assert(buffer[$-1] == 0);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1650 // Float number is well-formed. Check suffixes and do conversion.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1651 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1652 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1653 case 'f', 'F':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1654 t.type = TOK.Float32;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1655 t.float_ = strtof(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1656 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1657 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1658 case 'L':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1659 t.type = TOK.Float80;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1660 t.real_ = strtold(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1661 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1662 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1663 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1664 t.type = TOK.Float64;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1665 t.double_ = strtod(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1666 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1667 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1668 if (*p == 'i')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1669 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1670 ++p;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1671 t.type += 3; // Switch to imaginary counterpart.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1672 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1673 if (getErrno == ERANGE)
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1674 error(MID.OverflowFloatNumber);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1675 t.end = p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1676 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1677
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1678 /// Scan special token: #line Integer [Filespec] EndOfLine
360 b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence(). aziz parents: 350 diff changeset	1679 void scanSpecialTokenSequence(ref Token t)
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1680 {
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1681 assert(*p == '#');
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1682
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1683 t.type = TOK.HashLine;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1684
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1685 MID mid;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1686
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1687 ++p;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1688 if (p[0] != 'l' \|\| p[1] != 'i' \|\| p[2] != 'n' \|\| p[3] != 'e')
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1689 {
370 ae4afb66768f - Renamed findModule() to findModulePath(). aziz parents: 367 diff changeset	1690 mid = MID.ExpectedIdentifierSTLine;
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1691 goto Lerr;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1692 }
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1693 p += 3;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1694
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1695 // TODO: #line58"path/file" is legal. Require spaces?
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1696 // State.Space could be used for that purpose.
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1697 enum State
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1698 { /+Space,+/ Integer, Filespec, End }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1699
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1700 State state = State.Integer;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1701
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1702 Loop:
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1703 while (1)
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1704 {
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1705 switch (*++p)
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1706 {
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1707 case '\r':
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1708 if (p[1] == '\n')
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1709 ++p;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1710 case '\n', 0, _Z_:
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1711 break Loop;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1712 case LS[0]:
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1713 if (p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2]))
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1714 {
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1715 ++p; ++p;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1716 break Loop;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1717 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1718 goto default;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1719 default:
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1720 if (isspace(*p))
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1721 continue;
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1722 if (state == State.Integer)
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1723 {
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1724 if (!isdigit(*p))
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1725 {
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1726 mid = MID.ExpectedIntegerAfterSTLine;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1727 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1728 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1729 t.line_num = new Token;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1730 scan(*t.line_num);
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1731 if (t.line_num.type != TOK.Int32 && t.line_num.type != TOK.Uint32)
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1732 {
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1733 mid = MID.ExpectedIntegerAfterSTLine;
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1734 goto Lerr;
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1735 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1736 --p; // Go one back because scan() advanced p past the integer.
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1737 state = State.Filespec;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1738 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1739 else if (state == State.Filespec)
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1740 {
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1741 if (*p != '"')
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1742 {
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1743 mid = MID.ExpectedFilespec;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1744 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1745 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1746 t.line_filespec = new Token;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1747 t.line_filespec.start = p;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1748 t.line_filespec.type = TOK.Filespec;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1749 while (1)
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1750 {
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1751 switch (*++p)
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1752 {
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1753 case '"':
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1754 break;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1755 case LS[0]:
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1756 if (!(p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2])))
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1757 goto default;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1758 case '\r', '\n', 0, _Z_:
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1759 mid = MID.UnterminatedFilespec;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1760 t.line_filespec.end = p;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1761 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1762 default:
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1763 if (*p & 128)
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1764 decodeUTF8();
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1765 continue;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1766 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1767 break; // Exit loop.
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1768 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1769 auto start = t.line_filespec.start +1; // +1 skips '"'
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1770 t.line_filespec.str = start[0 .. p - start];
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1771 t.line_filespec.end = p + 1;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1772 state = State.End;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1773 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1774 else/+ if (state == State.End)+/
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1775 {
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1776 mid = MID.UnterminatedSpecialToken;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1777 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1778 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1779 }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1780 }
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1781
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1782 if (state == State.Integer)
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1783 {
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1784 mid = MID.ExpectedIntegerAfterSTLine;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1785 goto Lerr;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1786 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1787
371 01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1788 this.loc_old = this.loc;
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1789 this.loc_hline = t.line_num.uint_ - 1;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1790 if (t.line_filespec)
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1791 this.fileName = t.line_filespec.str;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1792 t.end = p;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1793
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1794 return;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1795 Lerr:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1796 t.end = p;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1797 error(mid);
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1798 }
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1799
371 01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1800 uint errorLoc()
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1801 {
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1802 // ∆loc + line_num_of(#line)
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1803 return this.loc - this.loc_old + this.loc_hline;
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1804 }
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1805
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1806 dchar decodeUTF8()
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1807 {
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1808 assert(*p & 128, "check for ASCII char before calling decodeUTF8().");
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1809 size_t idx;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1810 dchar d;
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1811 try
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1812 {
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1813 d = std.utf.decode(p[0 .. end-p], idx);
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1814 p += idx -1;
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1815 }
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1816 catch (UtfException e)
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1817 {
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1818 error(MID.InvalidUTF8Sequence);
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1819 // Skip to next valid utf-8 sequence
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1820 while (p < end && UTF8stride[*++p] != 0xFF) {}
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1821 --p;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1822 assert(p < end);
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1823 }
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1824 return d;
0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1825 }
0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1826
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1827 void loadKeywords()
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1828 {
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1829 foreach(k; keywords)
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1830 idtable[k.str] = k;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1831 }
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1832 /+ // Not needed anymore because tokens are stored in a linked list.
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1833 struct State
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1834 {
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1835 Lexer lexer;
209 921d15b1beeb - Added members token and loc to State. aziz parents: 207 diff changeset	1836 Token token;
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1837 char* scanPointer;
209 921d15b1beeb - Added members token and loc to State. aziz parents: 207 diff changeset	1838 int loc;
212 c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1839 string fileName;
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1840 size_t errorLen;
212 c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1841 static State opCall(Lexer lx)
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1842 {
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1843 State s;
212 c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1844 s.lexer = lx;
c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1845 s.token = lx.token;
c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1846 s.scanPointer = lx.p;
c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1847 s.loc = lx.loc;
c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1848 s.fileName = lx.fileName;
c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1849 s.errorLen = lx.errors.length;
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1850 return s;
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1851 }
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1852 void restore()
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1853 {
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1854 lexer.p = scanPointer;
209 921d15b1beeb - Added members token and loc to State. aziz parents: 207 diff changeset	1855 lexer.token = token;
921d15b1beeb - Added members token and loc to State. aziz parents: 207 diff changeset	1856 lexer.loc = loc;
212 c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1857 lexer.fileName = fileName;
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1858 lexer.errors = lexer.errors[0..errorLen];
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1859 }
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1860 }
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1861
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1862 State getState()
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1863 {
212 c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1864 return State(this);
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1865 }
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1866 +/
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1867
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1868 private void scanNext(ref Token* t)
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	1869 {
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1870 assert(t !is null);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1871 if (t.next)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1872 t = t.next;
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	1873 else if (t != this.tail)
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	1874 {
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1875 Token* new_t = new Token;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1876 scan(*new_t);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1877 new_t.prev = t;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1878 t.next = new_t;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1879 t = new_t;
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	1880 }
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1881 }
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1882
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1883 void peek(ref Token* t)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1884 {
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1885 scanNext(t);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1886 }
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1887
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1888 TOK nextToken()
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1889 {
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1890 scanNext(this.token);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1891 return this.token.type;
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	1892 }
511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	1893
66 8e84db78ad55 - Added support for variadic arguments in error messages. aziz parents: 65 diff changeset	1894 void error(MID id, ...)
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1895 {
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1896 // if (reportErrors)
371 01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1897 errors ~= new Information(InfoType.Lexer, id, this.errorLoc, arguments(_arguments, _argptr));
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1898 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1899
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	1900 unittest
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	1901 {
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1902 writefln("Testing method Lexer.peek()");
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	1903 string sourceText = "unittest { }";
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	1904 auto lx = new Lexer(sourceText, null);
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	1905
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	1906 Token* next = lx.head;
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	1907 lx.peek(next);
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	1908 assert(next.type == TOK.Unittest);
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	1909 lx.peek(next);
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	1910 assert(next.type == TOK.LBrace);
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	1911 lx.peek(next);
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	1912 assert(next.type == TOK.RBrace);
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	1913 lx.peek(next);
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	1914 assert(next.type == TOK.EOF);
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	1915 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	1916
306 9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	1917 Token* getTokens()
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	1918 {
306 9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	1919 while (nextToken() != TOK.EOF)
9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	1920 {}
9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	1921 return head;
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	1922 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1923
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	1924 static bool isNonReservedIdentifier(char[] ident)
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	1925 {
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1926 if (ident.length == 0)
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1927 return false;
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1928
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	1929 static Identifier[string] reserved_ids_table;
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	1930 if (reserved_ids_table is null)
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	1931 foreach(k; keywords)
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	1932 reserved_ids_table[k.str] = k;
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	1933
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1934 size_t idx = 1; // Index to the 2nd character in ident.
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1935 dchar isFirstCharUniAlpha()
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1936 {
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1937 idx = 0;
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1938 // NB: decode() could throw an Exception which would be
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1939 // caught by the next try-catch-block.
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1940 return isUniAlpha(std.utf.decode(ident, idx));
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1941 }
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1942
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	1943 try
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1944 {
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1945 if (isidbeg(ident[0]) \|\|
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1946 ident[0] & 128 && isFirstCharUniAlpha())
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1947 {
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1948 foreach (dchar c; ident[idx..$])
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1949 if (!isident(c) && !isUniAlpha(c))
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1950 return false;
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1951 }
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1952 }
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1953 catch (Exception)
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	1954 return false;
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	1955
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	1956 return !(ident in reserved_ids_table);
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	1957 }
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	1958
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1959 private void encodeUTF8(inout char[] str, dchar d)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1960 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1961 char[6] b;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1962 assert(d > 0x7F, "check for ASCII char before calling encodeUTF8().");
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1963 if (d < 0x800)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1964 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1965 b[0] = 0xC0 \| (d >> 6);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1966 b[1] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1967 str ~= b[0..2];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1968 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1969 else if (d < 0x10000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1970 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1971 b[0] = 0xE0 \| (d >> 12);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1972 b[1] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1973 b[2] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1974 str ~= b[0..3];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1975 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1976 else if (d < 0x200000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1977 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1978 b[0] = 0xF0 \| (d >> 18);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1979 b[1] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1980 b[2] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1981 b[3] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1982 str ~= b[0..4];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1983 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1984 else if (d < 0x4000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1985 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1986 b[0] = 0xF8 \| (d >> 24);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1987 b[1] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1988 b[2] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1989 b[3] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1990 b[4] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1991 str ~= b[0..5];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1992 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1993 else if (d < 0x80000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1994 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1995 b[0] = 0xFC \| (d >> 30);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1996 b[1] = 0x80 \| ((d >> 24) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1997 b[2] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1998 b[3] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1999 b[4] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2000 b[5] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2001 str ~= b[0..6];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2002 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2003 else
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2004 error(MID.InvalidUnicodeCharacter);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2005 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	2006 }
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2007
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2008 unittest
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2009 {
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2010 writefln("Testing Lexer.");
40 9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2011 string[] toks = [
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2012 ">", ">=", ">>", ">>=", ">>>", ">>>=", "<", "<=", "<>",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2013 "<>=", "<<", "<<=", "!", "!<", "!>", "!<=", "!>=", "!<>",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2014 "!<>=", ".", "..", "...", "&", "&&", "&=", "+", "++",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2015 "+=", "-", "--", "-=", "=", "==", "~", "~=", "*",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2016 "*=", "/", "/=", "^", "^=", "%", "%=", "(", ")",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2017 "[", "]", "{", "}", ":", ";", "?", ",", "$"
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2018 ];
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2019
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2020 char[] src;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2021
40 9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2022 foreach (op; toks)
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2023 src ~= op ~ " ";
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2024
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2025 auto lx = new Lexer(src, "");
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2026 auto token = lx.getTokens();
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2027
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2028 uint i;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2029 assert(token == lx.head);
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2030 token = token.next;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2031 do
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2032 {
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2033 assert(i < toks.length);
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2034 assert(token.srcText == toks[i], std.string.format("Scanned '%s' but expected '%s'", token.srcText, toks[i]));
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2035 ++i;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2036 token = token.next;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2037 } while (token.type != TOK.EOF)
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	2038 }
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2039
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2040 unittest
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2041 {
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2042 // Numbers unittest
68 7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2043 // 0L 0ULi 0_L 0_UL 0x0U 0x0p2 0_Fi 0_e2 0_F 0_i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2044 // 0u 0U 0uL 0UL 0L 0LU 0Lu
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2045 // 0Li 0f 0F 0fi 0Fi 0i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2046 // 0b_1_LU 0b1000u
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2047 // 0x232Lu
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2048 }
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2049
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2050 /// ASCII character properties table.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2051 static const int ptable[256] = [
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2052 0, 0, 0, 0, 0, 0, 0, 0, 0,32, 0,32,32, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2053 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2054 32, 0, 0x2200, 0, 0, 0, 0, 0x2700, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2055 7, 7, 7, 7, 7, 7, 7, 7, 6, 6, 0, 0, 0, 0, 0, 0x3f00,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2056 0,12,12,12,12,12,12, 8, 8, 8, 8, 8, 8, 8, 8, 8,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2057 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0, 0x5c00, 0, 0,16,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2058 0, 0x70c, 0x80c,12,12,12, 0xc0c, 8, 8, 8, 8, 8, 8, 8, 0xa08, 8,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2059 8, 8, 0xd08, 8, 0x908, 8, 0xb08, 8, 8, 8, 8, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2060 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2061 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2062 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2063 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2064 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2065 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2066 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2067 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2068 ];
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2069
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2070 enum CProperty
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2071 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2072 Octal = 1,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2073 Digit = 1<<1,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2074 Hex = 1<<2,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2075 Alpha = 1<<3,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2076 Underscore = 1<<4,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2077 Whitespace = 1<<5
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2078 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2079
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2080 const uint EVMask = 0xFF00; // Bit mask for escape value
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2081
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2082 private alias CProperty CP;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2083 int isoctal(char c) { return ptable[c] & CP.Octal; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2084 int isdigit(char c) { return ptable[c] & CP.Digit; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2085 int ishexad(char c) { return ptable[c] & CP.Hex; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2086 int isalpha(char c) { return ptable[c] & CP.Alpha; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2087 int isalnum(char c) { return ptable[c] & (CP.Alpha \| CP.Digit); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2088 int isidbeg(char c) { return ptable[c] & (CP.Alpha \| CP.Underscore); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2089 int isident(char c) { return ptable[c] & (CP.Alpha \| CP.Underscore \| CP.Digit); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2090 int isspace(char c) { return ptable[c] & CP.Whitespace; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2091 int char2ev(char c) { return ptable[c] >> 8; /(ptable[c] & EVMask) >> 8;/ }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2092
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2093 version(gen_ptable)
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2094 static this()
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2095 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2096 alias ptable p;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2097 // Initialize character properties table.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2098 for (int i; i < p.length; ++i)
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2099 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2100 p[i] = 0;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2101 if ('0' <= i && i <= '7')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2102 p[i] \|= CP.Octal;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2103 if ('0' <= i && i <= '9')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2104 p[i] \|= CP.Digit;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2105 if (isdigit(i) \|\| 'a' <= i && i <= 'f' \|\| 'A' <= i && i <= 'F')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2106 p[i] \|= CP.Hex;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2107 if ('a' <= i && i <= 'z' \|\| 'A' <= i && i <= 'Z')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2108 p[i] \|= CP.Alpha;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2109 if (i == '_')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2110 p[i] \|= CP.Underscore;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2111 if (i == ' ' \|\| i == '\t' \|\| i == '\v' \|\| i == '\f')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2112 p[i] \|= CP.Whitespace;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2113 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2114 // Store escape sequence values in second byte.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2115 assert(CProperty.max <= ubyte.max, "character property flags and escape value byte overlap.");
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2116 p['\''] \|= 39 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2117 p['"'] \|= 34 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2118 p['?'] \|= 63 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2119 p['\\'] \|= 92 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2120 p['a'] \|= 7 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2121 p['b'] \|= 8 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2122 p['f'] \|= 12 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2123 p['n'] \|= 10 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2124 p['r'] \|= 13 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2125 p['t'] \|= 9 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2126 p['v'] \|= 11 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2127 // Print a formatted array literal.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2128 char[] array = "[\n";
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2129 for (int i; i < p.length; ++i)
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2130 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2131 int c = p[i];
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2132 array ~= std.string.format(c>255?" 0x%x,":"%2d,", c, ((i+1) % 16) ? "":"\n");
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2133 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2134 array[$-2..$] = "\n]";
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2135 writefln(array);
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2136 }

Mercurial > projects > dil

annotate trunk/src/dil/Lexer.d @ 389:c4bfceab7246