projects/dil: trunk/src/dil/Lexer.d annotate

annotate trunk/src/dil/Lexer.d @ 412:fb31af0fda73

Added struct Location, and token2LocTable to Lexer. The token2LocTable member could be used to give the column of where a parser or lexer error occurred. And maybe it could be useful for statistics, too.

author	Aziz K?ksal <aziz.koeksal@gmail.com>
date	Tue, 25 Sep 2007 16:44:27 +0200
parents	cca83c0c00fd
children	0fd78fdcb982

rev	line source
0 8ba2570de175 Initial import. aziz parents: diff changeset	1 /++
8ba2570de175 Initial import. aziz parents: diff changeset	2 Author: Aziz Köksal
249 32d354584b28 - Upgraded license notices to GPL3. aziz parents: 239 diff changeset	3 License: GPL3
0 8ba2570de175 Initial import. aziz parents: diff changeset	4 +/
326 4a7359b88c11 - Added package 'dil' to module declarations. aziz parents: 325 diff changeset	5 module dil.Lexer;
327 a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	6 import dil.Token;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	7 import dil.Information;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	8 import dil.Keywords;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	9 import dil.Identifier;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	10 import dil.Messages;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	11 import dil.HtmlEntities;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	12 import dil.Settings;
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	13 import tango.stdc.stdlib : strtof, strtod, strtold;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	14 import tango.stdc.errno : errno, ERANGE;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	15 import tango.stdc.time : time_t, time, ctime;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	16 import tango.stdc.string : strlen;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	17 import std.utf;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	18 import std.uni;
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	19 import common;
0 8ba2570de175 Initial import. aziz parents: diff changeset	20
11 dffcdaa7c47a - Added Unicode line and paragraph separators. aziz parents: 10 diff changeset	21 const char[3] LS = \u2028;
dffcdaa7c47a - Added Unicode line and paragraph separators. aziz parents: 10 diff changeset	22 const char[3] PS = \u2029;
dffcdaa7c47a - Added Unicode line and paragraph separators. aziz parents: 10 diff changeset	23
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	24 const dchar LSd = 0x2028;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	25 const dchar PSd = 0x2029;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	26
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	27 const uint _Z_ = 26; /// Control+Z
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	28
412 fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	29 struct Location
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	30 {
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	31 size_t lineNum;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	32 char* filePath;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	33
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	34 static Location opCall(size_t lineNum, typeof(filePath) filePath)
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	35 {
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	36 Location l;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	37 l.lineNum = lineNum;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	38 l.filePath = filePath;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	39 return l;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	40 }
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	41 }
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	42
0 8ba2570de175 Initial import. aziz parents: diff changeset	43 class Lexer
8ba2570de175 Initial import. aziz parents: diff changeset	44 {
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	45 Token* head; /// The head of the doubly linked token list.
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	46 Token* tail; /// The tail of the linked list. Set in scan().
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	47 Token* token; /// Points to the current token in the token list.
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	48 string text;
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	49 char* p; /// Points to the current character in the source text.
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	50 char* end; /// Points one character past the end of the source text.
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	51
371 01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	52 uint loc = 1; /// Actual line of code.
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	53
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	54 uint loc_old; /// Store actual line number when #line token is parsed.
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	55 uint loc_hline; /// Line number set by #line.
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	56 private uint inTokenString; // > 0 if inside q{ }
17 9bd0bac79479 - Removed Whitespace from enum list. aziz parents: 16 diff changeset	57
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	58 char[] fileName;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	59
69 24db7c5522d5 - Added module Information for compiler messages like warnings, info and errors to the user. aziz parents: 68 diff changeset	60 Information[] errors;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	61
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	62 // bool reportErrors;
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	63
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	64 Identifier[string] idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	65
412 fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	66 version(token2LocTable)
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	67 /// Maps every token that starts a new line to a Location.
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	68 Location[Token*] token2LocTable;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	69
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	70 this(string text, string fileName)
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	71 {
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	72 this.fileName = fileName;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	73
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	74 this.text = text;
397 c99f8aeb7b4a Empty source files are handled correctly now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 396 diff changeset	75 if (text.length == 0 \|\| text[$-1] != 0)
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	76 {
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	77 this.text.length = this.text.length + 1;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	78 this.text[$-1] = 0;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	79 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	80
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	81 this.p = this.text.ptr;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	82 this.end = this.p + this.text.length;
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	83 // this.reportErrors = true;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	84 loadKeywords();
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	85
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	86 this.head = new Token;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	87 this.head.type = TOK.HEAD;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	88 this.token = this.head;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	89 scanShebang();
412 fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	90 version(token2LocTable)
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	91 {
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	92 // Add first token to table.
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	93 auto firstToken = this.head;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	94 peek(firstToken);
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	95 token2LocTable[firstToken] = Location(1, null);
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	96 }
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	97 }
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	98
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	99 ~this()
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	100 {
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	101 auto token = head.next;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	102 do
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	103 {
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	104 assert(token.type == TOK.EOF ? token == tail && token.next is null : 1);
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	105 delete token.prev;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	106 token = token.next;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	107 } while (token !is null)
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	108 delete tail;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	109 }
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	110
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	111 void scanShebang()
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	112 {
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	113 if (*p == '#' && p[1] == '!')
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	114 {
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	115 Token* t = new Token;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	116 t.start = p;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	117 t.type = TOK.Shebang;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	118 ++p;
403 e7228859d865 Fixed the way newlines are handled after #! and #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 402 diff changeset	119 assert(*p == '!');
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	120 while (1)
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	121 {
403 e7228859d865 Fixed the way newlines are handled after #! and #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 402 diff changeset	122 t.end = ++p;
e7228859d865 Fixed the way newlines are handled after #! and #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 402 diff changeset	123 switch (*p)
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	124 {
403 e7228859d865 Fixed the way newlines are handled after #! and #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 402 diff changeset	125 case '\n', '\r':
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	126 break;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	127 case 0, _Z_:
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	128 break;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	129 default:
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	130 if (*p & 128)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	131 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	132 auto c = decodeUTF8();
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	133 if (c == LSd \|\| c == PSd)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	134 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	135 }
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	136 continue;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	137 }
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	138 break; // Exit loop.
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	139 }
403 e7228859d865 Fixed the way newlines are handled after #! and #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 402 diff changeset	140 // Reset p. The newline will be scanned as whitespace in scan().
e7228859d865 Fixed the way newlines are handled after #! and #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 402 diff changeset	141 p = t.end;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	142 this.head.next = t;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	143 t.prev = this.head;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	144 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	145 }
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	146
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	147 void finalizeSpecialToken(ref Token t)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	148 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	149 assert(t.srcText[0..2] == "__");
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	150 switch (t.type)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	151 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	152 case TOK.FILE:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	153 t.str = this.fileName;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	154 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	155 case TOK.LINE:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	156 t.uint_ = this.loc;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	157 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	158 case TOK.DATE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	159 TOK.TIME,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	160 TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	161 time_t time_val;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	162 time(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	163 char* str = ctime(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	164 char[] time_str = str[0 .. strlen(str)];
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	165 switch (t.type)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	166 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	167 case TOK.DATE:
346 ce9e2e77743f - Fix: include one space when slicing time_str for __DATE__. aziz parents: 344 diff changeset	168 time_str = time_str[4..11] ~ time_str[20..24] ~ \0; break;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	169 case TOK.TIME:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	170 time_str = time_str[11..19] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	171 case TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	172 time_str = time_str[0..24] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	173 default: assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	174 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	175 t.str = time_str;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	176 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	177 case TOK.VENDOR:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	178 t.str = VENDOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	179 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	180 case TOK.VERSION:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	181 t.uint_ = VERSION_MAJOR*1000 + VERSION_MINOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	182 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	183 default:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	184 assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	185 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	186 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	187
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	188 public void scan(out Token t)
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	189 in
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	190 {
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	191 assert(text.ptr <= p && p < end);
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	192 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	193 out
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	194 {
397 c99f8aeb7b4a Empty source files are handled correctly now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 396 diff changeset	195 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type));
c99f8aeb7b4a Empty source files are handled correctly now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 396 diff changeset	196 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type));
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	197 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	198 body
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	199 {
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	200 // Scan whitespace.
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	201 auto pws = p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	202 while (1)
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	203 {
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	204 switch (*p)
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	205 {
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	206 case '\r':
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	207 if (p[1] == '\n')
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	208 ++p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	209 case '\n':
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	210 ++p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	211 ++loc;
412 fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	212 version(token2LocTable)
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	213 token2LocTable[&t] = Location(loc, null);
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	214 continue;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	215 case LS[0]:
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	216 if (p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2]))
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	217 {
412 fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	218 ++p; ++p;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	219 goto case '\n';
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	220 }
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	221 // goto default;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	222 default:
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	223 if (!isspace(*p))
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	224 break;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	225 ++p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	226 continue;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	227 }
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	228 break; // Exit loop.
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	229 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	230
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	231 if (p != pws)
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	232 t.ws = pws;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	233
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	234 // Scan token.
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	235 uint c = *p;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	236 {
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	237 t.start = p;
16 476e8e55c1d4 - Added Whitespace to the character properties table. aziz parents: 15 diff changeset	238
13 e5211758b63c - Added isidbeg() function. aziz parents: 12 diff changeset	239 if (isidbeg(c))
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	240 {
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	241 if (c == 'r' && p[1] == '"' && ++p)
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	242 return scanRawStringLiteral(t);
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	243 if (c == 'x' && p[1] == '"')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	244 return scanHexStringLiteral(t);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	245 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	246 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	247 if (c == 'q' && p[1] == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	248 return scanDelimitedStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	249 if (c == 'q' && p[1] == '{')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	250 return scanTokenStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	251 }
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	252 Lidentifier:
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	253 do
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	254 { c = *++p; }
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	255 while (isident(c) \|\| c & 128 && isUniAlpha(decodeUTF8()))
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	256
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	257 t.end = p;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	258
65 6c21ae79fbb3 - Renamed function Token.span to Token.srcText. aziz parents: 64 diff changeset	259 string str = t.srcText;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	260 Identifier* id = str in idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	261
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	262 if (!id)
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	263 {
327 a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	264 idtable[str] = Identifier(TOK.Identifier, str);
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	265 id = str in idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	266 }
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	267 assert(id);
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	268 t.type = id.type;
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	269 if (t.type == TOK.Identifier)
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	270 return;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	271 if (t.type == TOK.EOF)
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	272 {
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	273 t.type = TOK.EOF;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	274 t.end = p;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	275 tail = &t;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	276 assert(t.srcText == "__EOF__");
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	277 }
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	278 else if (t.isSpecialToken)
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	279 finalizeSpecialToken(t);
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	280 return;
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	281 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	282
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	283 if (isdigit(c))
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	284 return scanNumber(t);
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	285
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	286 if (c == '/')
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	287 {
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	288 c = *++p;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	289 switch(c)
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	290 {
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	291 case '=':
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	292 ++p;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	293 t.type = TOK.DivAssign;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	294 t.end = p;
cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	295 return;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	296 case '+':
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	297 uint level = 1;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	298 while (1)
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	299 {
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	300 c = *++p;
42 c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	301 LswitchNC: // only jumped to from default case of next switch(c)
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	302 switch (c)
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	303 {
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	304 case '\r':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	305 if (p[1] == '\n')
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	306 ++p;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	307 case '\n':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	308 ++loc;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	309 continue;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	310 case 0, _Z_:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	311 error(MID.UnterminatedNestedComment);
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	312 goto LreturnNC;
42 c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	313 default:
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	314 }
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	315
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	316 c <<= 8;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	317 c \|= *++p;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	318 switch (c)
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	319 {
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	320 case 0x2F2B: // /+
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	321 ++level;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	322 continue;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	323 case 0x2B2F: // +/
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	324 if (--level == 0)
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	325 {
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	326 ++p;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	327 LreturnNC:
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	328 t.type = TOK.Comment;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	329 t.end = p;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	330 return;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	331 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	332 continue;
42 c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	333 case 0xE280: // LS[0..1] \|\| PS[0..1]
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	334 if (p[1] == LS[2] \|\| p[1] == PS[2])
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	335 {
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	336 ++loc;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	337 ++p;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	338 }
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	339 continue;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	340 default:
42 c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	341 c &= char.max;
c6a9974a6e3c - Optimized scanner of nested comments. aziz parents: 41 diff changeset	342 goto LswitchNC;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	343 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	344 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	345 case '*':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	346 while (1)
7 07e45c06a024 - Parsing nested comments correctly now. aziz parents: 5 diff changeset	347 {
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	348 c = *++p;
1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	349 LswitchBC: // only jumped to from default case of next switch(c)
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	350 switch (c)
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	351 {
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	352 case '\r':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	353 if (p[1] == '\n')
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	354 ++p;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	355 case '\n':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	356 ++loc;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	357 continue;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	358 case 0, _Z_:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	359 error(MID.UnterminatedBlockComment);
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	360 goto LreturnBC;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	361 default:
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	362 }
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	363
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	364 c <<= 8;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	365 c \|= *++p;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	366 switch (c)
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	367 {
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	368 case 0x2A2F: // */
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	369 ++p;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	370 LreturnBC:
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	371 t.type = TOK.Comment;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	372 t.end = p;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	373 return;
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	374 case 0xE280: // LS[0..1] \|\| PS[0..1]
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	375 if (p[1] == LS[2] \|\| p[1] == PS[2])
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	376 {
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	377 ++loc;
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	378 ++p;
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	379 }
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	380 continue;
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	381 default:
2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	382 c &= char.max;
43 1845c23dd056 - Matched some parts of the scanner of block comments to the scanner of nested comments. aziz parents: 42 diff changeset	383 goto LswitchBC;
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	384 }
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	385 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	386 assert(0);
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	387 case '/':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	388 while (1)
7 07e45c06a024 - Parsing nested comments correctly now. aziz parents: 5 diff changeset	389 {
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	390 c = *++p;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	391 switch (c)
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	392 {
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	393 case '\r':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	394 if (p[1] == '\n')
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	395 ++p;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	396 case '\n':
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	397 case 0, _Z_:
11 dffcdaa7c47a - Added Unicode line and paragraph separators. aziz parents: 10 diff changeset	398 break;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	399 case LS[0]:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	400 if (p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2]))
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	401 break;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	402 continue;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	403 default:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	404 continue;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	405 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	406 t.type = TOK.Comment;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	407 t.end = p;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	408 return;
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	409 }
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	410 default:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	411 t.type = TOK.Div;
10 3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	412 t.end = p;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	413 return;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	414 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	415 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	416
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	417 switch (c)
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	418 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	419 case '\'':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	420 return scanCharacterLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	421 case '`':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	422 return scanRawStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	423 case '"':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	424 return scanNormalStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	425 case '\\':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	426 char[] buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	427 do
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	428 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	429 c = scanEscapeSequence();
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	430 if (c < 128)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	431 buffer ~= c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	432 else
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	433 encodeUTF8(buffer, c);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	434 } while (*p == '\\')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	435 buffer ~= 0;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	436 t.type = TOK.String;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	437 t.str = buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	438 t.end = p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	439 return;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	440 case '>': /* > >= >> >>= >>> >>>= */
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	441 c = *++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	442 switch (c)
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	443 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	444 case '=':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	445 t.type = TOK.GreaterEqual;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	446 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	447 case '>':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	448 if (p[1] == '>')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	449 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	450 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	451 if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	452 { ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	453 t.type = TOK.URShiftAssign;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	454 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	455 else
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	456 t.type = TOK.URShift;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	457 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	458 else if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	459 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	460 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	461 t.type = TOK.RShiftAssign;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	462 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	463 else
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	464 t.type = TOK.RShift;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	465 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	466 default:
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	467 t.type = TOK.Greater;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	468 goto Lcommon2;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	469 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	470 assert(0);
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	471 case '<': /* < <= <> <>= << <<= */
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	472 c = *++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	473 switch (c)
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	474 {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	475 case '=':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	476 t.type = TOK.LessEqual;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	477 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	478 case '<':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	479 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	480 ++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	481 t.type = TOK.LShiftAssign;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	482 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	483 else
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	484 t.type = TOK.LShift;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	485 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	486 case '>':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	487 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	488 ++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	489 t.type = TOK.LorEorG;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	490 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	491 else
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	492 t.type = TOK.LorG;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	493 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	494 default:
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	495 t.type = TOK.Less;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	496 goto Lcommon2;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	497 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	498 assert(0);
37 7f3bcb97d017 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 36 diff changeset	499 case '!': /* ! !< !> !<= !>= !<> !<>= */
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	500 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	501 switch (c)
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	502 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	503 case '<':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	504 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	505 if (c == '>')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	506 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	507 if (p[1] == '=') {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	508 ++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	509 t.type = TOK.Unordered;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	510 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	511 else
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	512 t.type = TOK.UorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	513 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	514 else if (c == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	515 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	516 t.type = TOK.UorG;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	517 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	518 else {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	519 t.type = TOK.UorGorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	520 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	521 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	522 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	523 case '>':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	524 if (p[1] == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	525 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	526 ++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	527 t.type = TOK.UorL;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	528 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	529 else
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	530 t.type = TOK.UorLorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	531 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	532 case '=':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	533 t.type = TOK.NotEqual;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	534 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	535 default:
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	536 t.type = TOK.Not;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	537 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	538 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	539 assert(0);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	540 case '.': /* . .[0-9] .. ... */
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	541 if (p[1] == '.')
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	542 {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	543 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	544 if (p[1] == '.') {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	545 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	546 t.type = TOK.Ellipses;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	547 }
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	548 else
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	549 t.type = TOK.Slice;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	550 }
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	551 else if (isdigit(p[1]))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	552 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	553 return scanReal(t);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	554 }
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	555 else
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	556 t.type = TOK.Dot;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	557 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	558 case '\|': /* \| \|\| \|= */
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	559 c = *++p;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	560 if (c == '=')
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	561 t.type = TOK.OrAssign;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	562 else if (c == '\|')
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	563 t.type = TOK.OrLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	564 else {
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	565 t.type = TOK.OrBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	566 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	567 }
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	568 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	569 case '&': /* & && &= */
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	570 c = *++p;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	571 if (c == '=')
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	572 t.type = TOK.AndAssign;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	573 else if (c == '&')
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	574 t.type = TOK.AndLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	575 else {
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	576 t.type = TOK.AndBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	577 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	578 }
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	579 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	580 case '+': /* + ++ += */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	581 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	582 if (c == '=')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	583 t.type = TOK.PlusAssign;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	584 else if (c == '+')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	585 t.type = TOK.PlusPlus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	586 else {
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	587 t.type = TOK.Plus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	588 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	589 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	590 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	591 case '-': /* - -- -= */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	592 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	593 if (c == '=')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	594 t.type = TOK.MinusAssign;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	595 else if (c == '-')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	596 t.type = TOK.MinusMinus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	597 else {
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	598 t.type = TOK.Minus;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	599 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	600 }
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	601 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	602 case '=': /* = == */
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	603 if (p[1] == '=') {
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	604 ++p;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	605 t.type = TOK.Equal;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	606 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	607 else
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	608 t.type = TOK.Assign;
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	609 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	610 case '~': /* ~ ~= */
27 43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	611 if (p[1] == '=') {
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	612 ++p;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	613 t.type = TOK.CatAssign;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	614 }
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	615 else
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	616 t.type = TOK.Tilde;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	617 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	618 case '': / * = /
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	619 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	620 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	621 t.type = TOK.MulAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	622 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	623 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	624 t.type = TOK.Mul;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	625 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	626 case '^': /* ^ ^= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	627 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	628 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	629 t.type = TOK.XorAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	630 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	631 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	632 t.type = TOK.Xor;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	633 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	634 case '%': /* % %= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	635 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	636 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	637 t.type = TOK.ModAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	638 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	639 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	640 t.type = TOK.Mod;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	641 goto Lcommon;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	642 // Single character tokens:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	643 case '(':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	644 t.type = TOK.LParen;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	645 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	646 case ')':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	647 t.type = TOK.RParen;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	648 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	649 case '[':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	650 t.type = TOK.LBracket;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	651 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	652 case ']':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	653 t.type = TOK.RBracket;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	654 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	655 case '{':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	656 t.type = TOK.LBrace;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	657 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	658 case '}':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	659 t.type = TOK.RBrace;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	660 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	661 case ':':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	662 t.type = TOK.Colon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	663 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	664 case ';':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	665 t.type = TOK.Semicolon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	666 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	667 case '?':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	668 t.type = TOK.Question;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	669 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	670 case ',':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	671 t.type = TOK.Comma;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	672 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	673 case '$':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	674 t.type = TOK.Dollar;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	675 Lcommon:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	676 ++p;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	677 Lcommon2:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	678 t.end = p;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	679 return;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	680 case '#':
360 b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence(). aziz parents: 350 diff changeset	681 return scanSpecialTokenSequence(t);
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	682 default:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	683 }
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	684
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	685 // Check for EOF
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	686 if (c == 0 \|\| c == _Z_)
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	687 {
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	688 assert(p == 0 \|\| p == _Z_);
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	689 t.type = TOK.EOF;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	690 t.end = p;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	691 tail = &t;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	692 assert(t.start == t.end);
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	693 return;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	694 }
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	695
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	696 if (c & 128)
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	697 {
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	698 c = decodeUTF8();
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	699 if (isUniAlpha(c))
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	700 goto Lidentifier;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	701 }
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	702
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	703 error(MID.IllegalCharacter, cast(dchar)c);
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	704
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	705 ++p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	706 t.type = TOK.Illegal;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	707 t.dchar_ = c;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	708 t.end = p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	709 return;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	710 }
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	711 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	712
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	713 void scanNormalStringLiteral(ref Token t)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	714 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	715 assert(*p == '"');
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	716 char[] buffer;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	717 t.type = TOK.String;
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	718 uint c;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	719 while (1)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	720 {
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	721 c = *++p;
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	722 switch (c)
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	723 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	724 case '"':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	725 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	726 Lreturn:
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	727 buffer ~= 0;
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	728 t.str = buffer;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	729 t.pf = scanPostfix();
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	730 t.end = p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	731 return;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	732 case '\\':
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	733 c = scanEscapeSequence();
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	734 --p;
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	735 if (c & 128)
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	736 encodeUTF8(buffer, c);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	737 else
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	738 break;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	739 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	740 case '\r':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	741 if (p[1] == '\n')
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	742 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	743 case '\n':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	744 ++loc;
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	745 c = '\n'; // Convert EndOfLine to \n.
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	746 break;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	747 case 0, _Z_:
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	748 error(MID.UnterminatedString);
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	749 goto Lreturn;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	750 default:
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	751 if (c & 128)
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	752 {
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	753 // char* begin = p;
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	754 c = decodeUTF8();
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	755 if (c == LSd \|\| c == PSd)
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	756 goto case '\n';
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	757
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	758 // We don't copy per pointer because we might include
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	759 // invalid, skipped utf-8 sequences. See decodeUTF8().
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	760 // ++p;
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	761 // buffer ~= begin[0 .. p - begin];
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	762 encodeUTF8(buffer, c);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	763 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	764 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	765 }
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	766 // Copy ASCII character.
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	767 buffer ~= c;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	768 }
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	769 assert(0);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	770 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	771
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	772 void scanCharacterLiteral(ref Token t)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	773 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	774 assert(*p == '\'');
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	775 MID id = MID.UnterminatedCharacterLiteral;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	776 ++p;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	777 TOK type = TOK.CharLiteral;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	778 switch (*p)
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	779 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	780 case '\\':
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	781 switch (p[1])
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	782 {
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	783 case 'u':
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	784 type = TOK.WCharLiteral; break;
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	785 case 'U':
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	786 type = TOK.DCharLiteral; break;
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	787 default:
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	788 }
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	789 t.dchar_ = scanEscapeSequence();
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	790 break;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	791 case '\'':
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	792 ++p;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	793 id = MID.EmptyCharacterLiteral;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	794 case '\n', '\r', 0, _Z_:
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	795 goto Lerr;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	796 default:
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	797 uint c = *p;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	798 if (c & 128)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	799 {
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	800 c = decodeUTF8();
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	801 if (c == LSd \|\| c == PSd)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	802 goto Lerr;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	803 if (c <= 0xFFFF)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	804 type = TOK.WCharLiteral;
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	805 else
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	806 type = TOK.DCharLiteral;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	807 }
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	808 t.dchar_ = c;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	809 ++p;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	810 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	811
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	812 if (*p == '\'')
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	813 ++p;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	814 else
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	815 Lerr:
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	816 error(id);
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	817 t.type = type;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	818 t.end = p;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	819 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	820
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	821 char scanPostfix()
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	822 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	823 switch (*p)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	824 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	825 case 'c':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	826 case 'w':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	827 case 'd':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	828 return *p++;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	829 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	830 return 0;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	831 }
112 004d98df65af - Implemented parseInterfaceDeclaration(). aziz parents: 105 diff changeset	832 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	833 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	834
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	835 void scanRawStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	836 {
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	837 uint delim = *p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	838 assert(delim == '`' \|\| delim == '"' && p[-1] == 'r');
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	839 t.type = TOK.String;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	840 char[] buffer;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	841 uint c;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	842 while (1)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	843 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	844 c = *++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	845 switch (c)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	846 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	847 case '\r':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	848 if (p[1] == '\n')
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	849 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	850 case '\n':
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	851 c = '\n'; // Convert EndOfLine ('\r','\r\n','\n',LS,PS) to '\n'
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	852 ++loc;
52 f65a83c27638 - Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well. aziz parents: 51 diff changeset	853 break;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	854 case '`':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	855 case '"':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	856 if (c == delim)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	857 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	858 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	859 t.pf = scanPostfix();
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	860 Lreturn:
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	861 t.str = buffer ~ '\0';
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	862 t.end = p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	863 return;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	864 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	865 break;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	866 case 0, _Z_:
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	867 if (delim == 'r')
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	868 error(MID.UnterminatedRawString);
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	869 else
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	870 error(MID.UnterminatedBackQuoteString);
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	871 goto Lreturn;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	872 default:
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	873 if (c & 128)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	874 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	875 c = decodeUTF8();
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	876 if (c == LSd \|\| c == PSd)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	877 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	878 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	879 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	880 }
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	881 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	882 buffer ~= c; // copy character to buffer
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	883 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	884 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	885 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	886
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	887 void scanHexStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	888 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	889 assert(p[0] == 'x' && p[1] == '"');
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	890 t.type = TOK.String;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	891
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	892 uint c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	893 ubyte[] buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	894 ubyte h; // hex number
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	895 uint n; // number of hex digits
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	896
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	897 ++p;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	898 while (1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	899 {
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	900 c = *++p;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	901 switch (c)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	902 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	903 case '"':
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	904 ++p;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	905 if (n & 1)
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	906 error(MID.OddNumberOfDigitsInHexString);
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	907 t.pf = scanPostfix();
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	908 Lreturn:
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	909 buffer ~= 0;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	910 t.str = cast(string) buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	911 t.end = p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	912 return;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	913 case '\r':
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	914 if (p[1] == '\n')
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	915 ++p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	916 case '\n':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	917 ++loc;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	918 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	919 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	920 if (ishexad(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	921 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	922 if (c <= '9')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	923 c -= '0';
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	924 else if (c <= 'F')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	925 c -= 'A' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	926 else
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	927 c -= 'a' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	928
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	929 if (n & 1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	930 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	931 h <<= 4;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	932 h \|= c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	933 buffer ~= h;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	934 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	935 else
112 004d98df65af - Implemented parseInterfaceDeclaration(). aziz parents: 105 diff changeset	936 h = cast(ubyte)c;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	937 ++n;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	938 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	939 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	940 else if (isspace(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	941 continue;
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	942
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	943 if (c & 128)
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	944 {
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	945 c = decodeUTF8();
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	946 if (c == LSd \|\| c == PSd)
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	947 {
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	948 ++p; ++p;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	949 ++loc;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	950 continue;
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	951 }
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	952 }
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	953 else if (c == 0 \|\| c == _Z_)
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	954 {
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	955 error(MID.UnterminatedHexString);
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	956 t.pf = 0;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	957 goto Lreturn;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	958 }
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	959 error(MID.NonHexCharInHexString, cast(dchar)c);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	960 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	961 }
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	962 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	963 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	964
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	965 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	966 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	967 void scanDelimitedStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	968 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	969 assert(p[0] == 'q' && p[1] == '"');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	970 t.type = TOK.String;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	971
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	972 char[] buffer;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	973 dchar opening_delim = 0, // 0 if no nested delimiter or '[', '(', '<', '{'
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	974 closing_delim; // Will be ']', ')', '>', '},
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	975 // the first character of an identifier or
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	976 // any other Unicode/ASCII character.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	977 char[] str_delim; // Identifier delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	978 uint level = 1; // Counter for nestable delimiters.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	979
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	980 ++p; ++p; // Skip q"
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	981 uint c = *p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	982 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	983 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	984 case '(':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	985 opening_delim = c;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	986 closing_delim = ')'; // c + 1
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	987 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	988 case '[', '<', '{':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	989 opening_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	990 closing_delim = c + 2; // Get to closing counterpart. Feature of ASCII table.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	991 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	992 default:
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	993 dchar scanNewline()
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	994 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	995 switch (*p)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	996 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	997 case '\r':
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	998 if (p[1] == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	999 ++p;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1000 case '\n':
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1001 ++p;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1002 ++loc;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1003 return '\n';
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1004 case LS[0]:
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1005 if (p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2]))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1006 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1007 ++p; ++p; ++p;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1008 ++loc;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1009 return '\n';
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1010 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1011 default:
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1012 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1013 return 0;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1014 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1015
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1016 // Skip leading newlines:
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1017 while (scanNewline() != 0){}
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1018 assert(p != '\n' && p != '\r');
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1019 assert(!(*p == LS[0] && p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2])));
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1020
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1021 char* begin = p;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1022 c = *p;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1023 closing_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1024 // TODO: Check for non-printable characters?
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1025 if (c & 128)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1026 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1027 closing_delim = decodeUTF8();
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1028 if (!isUniAlpha(closing_delim))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1029 break; // Not an identifier.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1030 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1031 else if (!isidbeg(c))
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1032 break; // Not an identifier.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1033
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1034 // Parse Identifier + EndOfLine
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1035 do
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1036 { c = *++p; }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1037 while (isident(c) \|\| c & 128 && isUniAlpha(decodeUTF8()))
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1038 // Store identifier
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1039 str_delim = begin[0..p-begin];
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1040 // Scan newline
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1041 if (scanNewline() == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1042 --p; // Go back one because of "c = *++p;" in main loop.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1043 else
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1044 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1045 // TODO: error(MID.ExpectedNewlineAfterIdentDelim);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1046 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1047 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1048
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1049 bool checkStringDelim(char* p)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1050 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1051 assert(str_delim.length != 0);
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1052 if (buffer[$-1] == '\n' && // Last character copied to buffer must be '\n'.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1053 end-p >= str_delim.length && // Check remaining length.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1054 p[0..str_delim.length] == str_delim) // Compare.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1055 return true;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1056 return false;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1057 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1058
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1059 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1060 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1061 c = *++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1062 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1063 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1064 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1065 if (p[1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1066 ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1067 case '\n':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1068 c = '\n'; // Convert EndOfLine ('\r','\r\n','\n',LS,PS) to '\n'
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1069 ++loc;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1070 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1071 case 0, _Z_:
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1072 // TODO: error(MID.UnterminatedDelimitedString);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1073 goto Lreturn3;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1074 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1075 if (c & 128)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1076 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1077 auto begin = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1078 c = decodeUTF8();
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1079 if (c == LSd \|\| c == PSd)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1080 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1081 if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1082 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1083 if (str_delim.length)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1084 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1085 if (checkStringDelim(begin))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1086 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1087 p = begin + str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1088 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1089 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1090 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1091 else
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1092 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1093 assert(level == 1);
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1094 --level;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1095 goto Lreturn;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1096 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1097 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1098 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1099 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1100 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1101 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1102 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1103 if (c == opening_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1104 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1105 else if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1106 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1107 if (str_delim.length)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1108 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1109 if (checkStringDelim(p))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1110 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1111 p += str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1112 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1113 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1114 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1115 else if (--level == 0)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1116 goto Lreturn;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1117 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1118 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1119 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1120 buffer ~= c; // copy character to buffer
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1121 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1122 Lreturn: // Character delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1123 assert(c == closing_delim);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1124 assert(level == 0);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1125 ++p; // Skip closing delimiter.
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1126 Lreturn2: // String delimiter.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1127 if (*p == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1128 ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1129 // else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1130 // TODO: error(MID.ExpectedDblQuoteAfterDelim, str_delim.length ? str_delim : p[-1]);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1131
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1132 t.pf = scanPostfix();
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1133 Lreturn3: // Error.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1134 t.str = buffer ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1135 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1136 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1137
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1138 void scanTokenStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1139 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1140 assert(p[0] == 'q' && p[1] == '{');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1141 t.type = TOK.String;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1142
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1143 // A guard against changes to particular members:
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1144 // this.loc_old, this.loc_hline and this.fileName
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1145 ++inTokenString;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1146
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1147 uint loc = this.loc;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1148 uint level = 1;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1149
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1150 ++p; ++p; // Skip q{
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1151
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1152 auto prev_t = &t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1153 Token* token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1154 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1155 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1156 token = new Token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1157 scan(*token);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1158 // Save the tokens in a doubly linked list.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1159 // Could be useful for various tools.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1160 token.prev = prev_t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1161 prev_t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1162 prev_t = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1163 switch (token.type)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1164 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1165 case TOK.LBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1166 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1167 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1168 case TOK.RBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1169 if (--level == 0)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1170 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1171 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1172 t.next = null;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1173 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1174 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1175 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1176 case TOK.EOF:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1177 // TODO: error(MID.UnterminatedTokenString);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1178 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1179 t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1180 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1181 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1182 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1183 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1184 break; // Exit loop.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1185 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1186
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1187 assert(token.type == TOK.RBrace \|\| token.type == TOK.EOF);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1188 assert(token.type == TOK.RBrace && t.next is null \|\|
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1189 token.type == TOK.EOF && t.next !is null);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1190
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1191 char[] buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1192 // token points to } or EOF
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1193 if (token.type == TOK.EOF)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1194 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1195 t.end = token.start;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1196 buffer = t.srcText[2..$].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1197 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1198 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1199 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1200 // Assign to buffer before scanPostfix().
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1201 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1202 buffer = t.srcText[2..$-1].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1203 t.pf = scanPostfix();
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1204 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1205 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1206 // Convert EndOfLines to '\n'
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1207 if (loc != this.loc)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1208 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1209 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1210 uint i, j;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1211 for (; i < buffer.length; ++i)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1212 switch (buffer[i])
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1213 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1214 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1215 if (buffer[i+1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1216 ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1217 case '\n':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1218 buffer[j++] = '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1219 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1220 case LS[0]:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1221 auto b = buffer[i..$];
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1222 if (b[1] == LS[1] && (b[2] == LS[2] \|\| b[2] == PS[2]))
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1223 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1224 ++i; ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1225 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1226 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1227 // goto default;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1228 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1229 buffer[j++] = buffer[i]; // Copy character
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1230 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1231 buffer.length = j; // Adjust length
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1232 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1233 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1234 t.str = buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1235
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1236 --inTokenString;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1237 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1238 } // version(D2)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1239
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1240 dchar scanEscapeSequence()
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1241 {
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1242 assert(*p == '\\');
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1243 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1244 uint c = char2ev(*p);
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1245 if (c)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1246 {
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1247 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1248 return c;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1249 }
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1250 uint digits = 2;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1251
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1252 switch (*p)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1253 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1254 case 'x':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1255 c = 0;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1256 while (1)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1257 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1258 ++p;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1259 if (ishexad(*p))
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1260 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1261 c *= 16;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1262 if (*p <= '9')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1263 c += *p - '0';
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1264 else if (*p <= 'F')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1265 c += *p - 'A' + 10;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1266 else
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1267 c += *p - 'a' + 10;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1268
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1269 if (!--digits)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1270 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1271 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1272 break;
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1273 }
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1274 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1275 else
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1276 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1277 error(MID.InsufficientHexDigits);
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1278 break;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1279 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1280 }
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1281 if (!isValidDchar(c))
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1282 error(MID.InvalidUnicodeCharacter);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1283 break;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1284 case 'u':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1285 digits = 4;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1286 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1287 case 'U':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1288 digits = 8;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1289 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1290 default:
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1291 if (isoctal(*p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1292 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1293 c = 0;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1294 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1295 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1296 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1297 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1298 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1299 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1300 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1301 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1302 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1303 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1304 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1305 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1306 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1307 else if(*p == '&')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1308 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1309 if (isalpha(*++p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1310 {
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1311 auto begin = p;
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1312 while (isalnum(*++p))
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1313 {}
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1314
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1315 if (*p == ';')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1316 {
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1317 c = entity2Unicode(begin[0..p - begin]);
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1318 ++p; // Skip ;
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1319 if (c == 0xFFFF)
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1320 error(MID.UndefinedHTMLEntity, (begin-1)[0..p-(begin-1)]);
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1321 }
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1322 else
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1323 error(MID.UnterminatedHTMLEntity);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1324 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1325 else
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1326 error(MID.InvalidBeginHTMLEntity);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1327 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1328 else
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1329 {
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1330 dchar d = *p;
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1331 char[] str = `\`;
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1332 if (d & 128)
387 ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1333 encodeUTF8(str, decodeUTF8());
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1334 else
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1335 str ~= d;
387 ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1336 ++p;
ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1337 // TODO: check for unprintable character?
ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1338 error(MID.UndefinedEscapeSequence, str);
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1339 }
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1340 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1341
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1342 return c;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1343 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1344
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1345 /*
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1346 IntegerLiteral:= (Dec\|Hex\|Bin\|Oct)Suffix?
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1347 Dec:= (0\|[1-9][0-9_]*)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1348 Hex:= 0[xX] HexDigits
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1349 Bin:= 0[bB][01_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1350 Oct:= 0[0-7_]+
68 7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	1351 Suffix:= (L[uU]?\|[uU]L?)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1352 HexDigits:= [0-9a-zA-Z_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1353
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1354 Invalid: "0b_", "0x_", "._"
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1355 */
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1356 void scanNumber(ref Token t)
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1357 {
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1358 ulong ulong_;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1359 bool overflow;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1360 bool isDecimal;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1361 size_t digits;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1362
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1363 if (*p != '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1364 goto LscanInteger;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1365 ++p; // skip zero
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1366 // check for xX bB ...
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1367 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1368 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1369 case 'x','X':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1370 goto LscanHex;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1371 case 'b','B':
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1372 goto LscanBinary;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1373 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1374 if (p[1] == 'i')
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1375 goto LscanReal; // 0Li
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1376 break; // 0L
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1377 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1378 if (p[1] == '.')
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1379 break; // 0..
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1380 // 0.
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1381 case 'i','f','F', // Imaginary and float literal suffixes.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1382 'e', 'E': // Float exponent.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1383 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1384 default:
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1385 if (*p == '_')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1386 goto LscanOctal; // 0_
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1387 else if (isdigit(*p))
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1388 {
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1389 if (p == '8' \|\| p == '9')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1390 goto Loctal_hasDecimalDigits; // 08 or 09
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1391 else
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1392 goto Loctal_enter_loop; // 0[0-7]
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1393 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1394 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1395
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1396 // Number 0
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1397 assert(p[-1] == '0');
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1398 assert(p != '_' && !isdigit(p));
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1399 assert(ulong_ == 0);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1400 isDecimal = true;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1401 goto Lfinalize;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1402
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1403 LscanInteger:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1404 assert(p != 0 && isdigit(p));
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1405 isDecimal = true;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1406 goto Lenter_loop_int;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1407 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1408 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1409 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1410 continue;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1411 if (!isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1412 break;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1413 Lenter_loop_int:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1414 if (ulong_ < ulong.max/10 \|\| (ulong_ == ulong.max/10 && *p <= '5'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1415 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1416 ulong_ *= 10;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1417 ulong_ += *p - '0';
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1418 continue;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1419 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1420 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1421 overflow = true;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1422 while (isdigit(*++p)) {}
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1423 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1424 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1425
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1426 // The number could be a float, so check overflow below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1427 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1428 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1429 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1430 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1431 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1432 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1433 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1434 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1435 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1436 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1437 goto LscanReal;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1438 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1439 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1440
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1441 if (overflow)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1442 error(MID.OverflowDecimalNumber);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1443
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1444 assert((isdigit(p[-1]) \|\| p[-1] == '_') && !isdigit(p) && p != '_');
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1445 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1446
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1447 LscanHex:
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1448 assert(digits == 0);
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	1449 assert(p == 'x' \|\| p == 'X');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1450 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1451 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1452 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1453 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1454 if (!ishexad(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1455 break;
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1456 ++digits;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1457 ulong_ *= 16;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1458 if (*p <= '9')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1459 ulong_ += *p - '0';
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1460 else if (*p <= 'F')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1461 ulong_ += *p - 'A' + 10;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1462 else
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1463 ulong_ += *p - 'a' + 10;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1464 }
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1465
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	1466 assert(ishexad(p[-1]) \|\| p[-1] == '_' \|\| p[-1] == 'x' \|\| p[-1] == 'X');
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1467 assert(!ishexad(p) && p != '_');
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1468
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1469 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1470 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1471 case '.':
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1472 if (p[1] == '.')
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1473 break;
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1474 case 'p', 'P':
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1475 return scanHexReal(t);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1476 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1477 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1478
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1479 if (digits == 0)
59 3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	1480 error(MID.NoDigitsInHexNumber);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1481 else if (digits > 16)
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1482 error(MID.OverflowHexNumber);
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1483
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1484 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1485
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1486 LscanBinary:
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1487 assert(digits == 0);
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	1488 assert(p == 'b' \|\| p == 'B');
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1489 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1490 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1491 if (*++p == '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1492 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1493 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1494 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1495 }
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1496 else if (*p == '1')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1497 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1498 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1499 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1500 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1501 }
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1502 else if (*p == '_')
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1503 continue;
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1504 else
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1505 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1506 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1507
59 3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	1508 if (digits == 0)
3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	1509 error(MID.NoDigitsInBinNumber);
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1510 else if (digits > 64)
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1511 error(MID.OverflowBinaryNumber);
59 3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	1512
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	1513 assert(p[-1] == '0' \|\| p[-1] == '1' \|\| p[-1] == '_' \|\| p[-1] == 'b' \|\| p[-1] == 'B', p[-1] ~ "");
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1514 assert( !(p == '0' \|\| p == '1' \|\| *p == '_') );
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1515 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1516
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1517 LscanOctal:
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1518 assert(*p == '_');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1519 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1520 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1521 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1522 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1523 if (!isoctal(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1524 break;
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1525 Loctal_enter_loop:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1526 if (ulong_ < ulong.max/2 \|\| (ulong_ == ulong.max/2 && *p <= '1'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1527 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1528 ulong_ *= 8;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1529 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1530 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1531 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1532 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1533 overflow = true;
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1534 while (isoctal(*++p)) {}
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1535 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1536 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1537
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1538 bool hasDecimalDigits;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1539 if (isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1540 {
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1541 Loctal_hasDecimalDigits:
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1542 hasDecimalDigits = true;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1543 while (isdigit(*++p)) {}
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1544 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1545
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1546 // The number could be a float, so check errors below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1547 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1548 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1549 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1550 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1551 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1552 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1553 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1554 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1555 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1556 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1557 goto LscanReal;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1558 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1559 }
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1560
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1561 if (hasDecimalDigits)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1562 error(MID.OctalNumberHasDecimals);
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1563 if (overflow)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1564 error(MID.OverflowOctalNumber);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1565 // goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1566
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1567 Lfinalize:
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1568 enum Suffix
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1569 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1570 None = 0,
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1571 Unsigned = 1,
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1572 Long = 2
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1573 }
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1574
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1575 // Scan optional suffix: L, Lu, LU, u, uL, U or UL.
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1576 Suffix suffix;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1577 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1578 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1579 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1580 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1581 case 'L':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1582 if (suffix & Suffix.Long)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1583 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1584 suffix \|= Suffix.Long;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1585 ++p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1586 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1587 case 'u', 'U':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1588 if (suffix & Suffix.Unsigned)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1589 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1590 suffix \|= Suffix.Unsigned;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1591 ++p;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	1592 continue;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1593 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1594 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1595 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1596 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1597 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1598
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1599 // Determine type of Integer.
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1600 switch (suffix)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1601 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1602 case Suffix.None:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1603 if (ulong_ & 0x8000000000000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1604 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1605 if (isDecimal)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1606 error(MID.OverflowDecimalSign);
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1607 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1608 }
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1609 else if (ulong_ & 0xFFFFFFFF00000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1610 t.type = TOK.Int64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1611 else if (ulong_ & 0x80000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1612 t.type = isDecimal ? TOK.Int64 : TOK.Uint32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1613 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1614 t.type = TOK.Int32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1615 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1616 case Suffix.Unsigned:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1617 if (ulong_ & 0xFFFFFFFF00000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1618 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1619 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1620 t.type = TOK.Uint32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1621 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1622 case Suffix.Long:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1623 if (ulong_ & 0x8000000000000000)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1624 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1625 if (isDecimal)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1626 error(MID.OverflowDecimalSign);
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1627 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1628 }
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1629 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1630 t.type = TOK.Int64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1631 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1632 case Suffix.Unsigned \| Suffix.Long:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1633 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1634 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1635 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1636 assert(0);
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1637 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1638 t.ulong_ = ulong_;
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1639 t.end = p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1640 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1641 LscanReal:
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1642 scanReal(t);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1643 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1644 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1645
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1646 /*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1647 FloatLiteral:= Float[fFL]?i?
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1648 Float:= DecFloat \| HexFloat
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1649 DecFloat:= ([0-9][0-9_][.][0-9_]DecExponent?) \| [.][0-9][0-9_]DecExponent? \| [0-9][0-9_]DecExponent
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1650 DecExponent:= [eE][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1651 HexFloat:= 0[xX](HexDigits[.]HexDigits \| [.][0-9a-zA-Z]HexDigits? \| HexDigits)HexExponent
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1652 HexExponent:= [pP][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1653 */
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1654 void scanReal(ref Token t)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1655 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1656 if (*p == '.')
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1657 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1658 assert(p[1] != '.');
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1659 // This function was called by scan() or scanNumber().
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1660 while (isdigit(++p) \|\| p == '_') {}
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1661 }
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1662 else
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1663 // This function was called by scanNumber().
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1664 assert(delegate (){
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1665 switch (*p)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1666 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1667 case 'L':
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1668 if (p[1] != 'i')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1669 return false;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1670 case 'i', 'f', 'F', 'e', 'E':
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1671 return true;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1672 default:
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1673 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1674 return false;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1675 }()
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1676 );
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1677
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1678 // Scan exponent.
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1679 if (p == 'e' \|\| p == 'E')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1680 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1681 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1682 if (p == '-' \|\| p == '+')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1683 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1684 if (!isdigit(*p))
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1685 error(MID.FloatExpMustStartWithDigit);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1686 else
64 dd4c5a0e47dd - Improved while loop. aziz parents: 63 diff changeset	1687 while (isdigit(++p) \|\| p == '_') {}
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1688 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1689
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1690 // Copy whole number and remove underscores from buffer.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1691 char[] buffer = t.start[0..p-t.start].dup;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1692 uint j;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1693 foreach (c; buffer)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1694 if (c != '_')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1695 buffer[j++] = c;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1696 buffer.length = j; // Adjust length.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1697 buffer ~= 0; // Terminate for C functions.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1698
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1699 finalizeFloat(t, buffer);
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1700 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1701
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1702 void scanHexReal(ref Token t)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1703 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1704 assert(p == '.' \|\| p == 'p' \|\| *p == 'P');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1705 MID mid;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1706 if (*p == '.')
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1707 while (ishexad(++p) \|\| p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1708 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1709 // Decimal exponent is required.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1710 if (p != 'p' && p != 'P')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1711 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1712 mid = MID.HexFloatExponentRequired;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1713 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1714 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1715 // Scan exponent
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1716 assert(p == 'p' \|\| p == 'P');
393 fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	1717 ++p;
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	1718 if (p == '+' \|\| p == '-')
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	1719 ++p;
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	1720 if (!isdigit(*p))
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1721 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1722 mid = MID.HexFloatExpMustStartWithDigit;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1723 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1724 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1725 while (isdigit(++p) \|\| p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1726 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1727 // Copy whole number and remove underscores from buffer.
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1728 char[] buffer = t.start[0..p-t.start].dup;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1729 uint j;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1730 foreach (c; buffer)
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1731 if (c != '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1732 buffer[j++] = c;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1733 buffer.length = j; // Adjust length.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1734 buffer ~= 0; // Terminate for C functions.
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1735 finalizeFloat(t, buffer);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1736 return;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1737 Lerr:
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1738 t.type = TOK.Float32;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1739 t.end = p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1740 error(mid);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1741 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1742
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1743 void finalizeFloat(ref Token t, string buffer)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1744 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1745 assert(buffer[$-1] == 0);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1746 // Float number is well-formed. Check suffixes and do conversion.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1747 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1748 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1749 case 'f', 'F':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1750 t.type = TOK.Float32;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1751 t.float_ = strtof(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1752 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1753 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1754 case 'L':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1755 t.type = TOK.Float80;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1756 t.real_ = strtold(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1757 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1758 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1759 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1760 t.type = TOK.Float64;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1761 t.double_ = strtod(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1762 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1763 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1764 if (*p == 'i')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1765 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1766 ++p;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1767 t.type += 3; // Switch to imaginary counterpart.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1768 }
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	1769 if (errno() == ERANGE)
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1770 error(MID.OverflowFloatNumber);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1771 t.end = p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1772 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1773
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1774 /// Scan special token: #line Integer [Filespec] EndOfLine
360 b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence(). aziz parents: 350 diff changeset	1775 void scanSpecialTokenSequence(ref Token t)
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1776 {
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1777 assert(*p == '#');
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1778
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1779 t.type = TOK.HashLine;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1780
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1781 MID mid;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1782
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1783 ++p;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1784 if (p[0] != 'l' \|\| p[1] != 'i' \|\| p[2] != 'n' \|\| p[3] != 'e')
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1785 {
370 ae4afb66768f - Renamed findModule() to findModulePath(). aziz parents: 367 diff changeset	1786 mid = MID.ExpectedIdentifierSTLine;
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1787 goto Lerr;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1788 }
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1789 p += 3;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1790
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1791 // TODO: #line58"path/file" is legal. Require spaces?
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1792 // State.Space could be used for that purpose.
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1793 enum State
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1794 { /+Space,+/ Integer, Filespec, End }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1795
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1796 State state = State.Integer;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1797
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1798 Loop:
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1799 while (1)
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1800 {
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1801 switch (*++p)
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1802 {
403 e7228859d865 Fixed the way newlines are handled after #! and #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 402 diff changeset	1803 case LS[0]:
e7228859d865 Fixed the way newlines are handled after #! and #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 402 diff changeset	1804 if (!(p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2])))
e7228859d865 Fixed the way newlines are handled after #! and #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 402 diff changeset	1805 goto default;
e7228859d865 Fixed the way newlines are handled after #! and #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 402 diff changeset	1806 case '\r', '\n', 0, _Z_:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1807 break Loop;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1808 default:
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1809 if (isspace(*p))
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1810 continue;
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1811 if (state == State.Integer)
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1812 {
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1813 if (!isdigit(*p))
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1814 {
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1815 mid = MID.ExpectedIntegerAfterSTLine;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1816 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1817 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1818 t.line_num = new Token;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1819 scan(*t.line_num);
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1820 if (t.line_num.type != TOK.Int32 && t.line_num.type != TOK.Uint32)
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1821 {
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1822 mid = MID.ExpectedIntegerAfterSTLine;
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1823 goto Lerr;
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1824 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1825 --p; // Go one back because scan() advanced p past the integer.
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1826 state = State.Filespec;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1827 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1828 else if (state == State.Filespec)
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1829 {
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1830 if (*p != '"')
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1831 {
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1832 mid = MID.ExpectedFilespec;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1833 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1834 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1835 t.line_filespec = new Token;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1836 t.line_filespec.start = p;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1837 t.line_filespec.type = TOK.Filespec;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1838 while (1)
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1839 {
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1840 switch (*++p)
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1841 {
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1842 case '"':
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1843 break;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1844 case LS[0]:
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1845 if (!(p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2])))
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1846 goto default;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1847 case '\r', '\n', 0, _Z_:
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1848 mid = MID.UnterminatedFilespec;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1849 t.line_filespec.end = p;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1850 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1851 default:
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1852 if (*p & 128)
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1853 decodeUTF8();
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1854 continue;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1855 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1856 break; // Exit loop.
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1857 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1858 auto start = t.line_filespec.start +1; // +1 skips '"'
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1859 t.line_filespec.str = start[0 .. p - start];
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1860 t.line_filespec.end = p + 1;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1861 state = State.End;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1862 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1863 else/+ if (state == State.End)+/
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1864 {
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1865 mid = MID.UnterminatedSpecialToken;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1866 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1867 }
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1868 }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1869 }
403 e7228859d865 Fixed the way newlines are handled after #! and #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 402 diff changeset	1870 assert(p == '\r' \|\| p == '\n' \|\| p == 0 \|\| p == _Z_ \|\|
e7228859d865 Fixed the way newlines are handled after #! and #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 402 diff changeset	1871 *p == LS[0] && (p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2]))
e7228859d865 Fixed the way newlines are handled after #! and #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 402 diff changeset	1872 );
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1873
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1874 if (state == State.Integer)
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	1875 {
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	1876 mid = MID.ExpectedIntegerAfterSTLine;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1877 goto Lerr;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1878 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1879
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1880 // Evaluate #line only when not in token string.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1881 if (!inTokenString)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1882 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1883 this.loc_old = this.loc;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1884 this.loc_hline = t.line_num.uint_ - 1;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1885 if (t.line_filespec)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1886 this.fileName = t.line_filespec.str;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1887 }
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1888 t.end = p;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1889
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1890 return;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1891 Lerr:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1892 t.end = p;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	1893 error(mid);
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1894 }
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1895
371 01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1896 uint errorLoc()
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1897 {
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1898 // ∆loc + line_num_of(#line)
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1899 return this.loc - this.loc_old + this.loc_hline;
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1900 }
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	1901
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1902 dchar decodeUTF8()
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1903 {
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1904 assert(*p & 128, "check for ASCII char before calling decodeUTF8().");
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1905 size_t idx;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1906 dchar d;
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1907 try
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1908 {
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1909 d = std.utf.decode(p[0 .. end-p], idx);
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1910 p += idx -1;
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1911 }
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1912 catch (UtfException e)
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1913 {
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1914 error(MID.InvalidUTF8Sequence);
7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1915 // Skip to next valid utf-8 sequence
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1916 while (p < end && UTF8stride[*++p] != 0xFF) {}
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1917 --p;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1918 assert(p < end);
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1919 }
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1920 return d;
0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1921 }
0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	1922
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1923 void loadKeywords()
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1924 {
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1925 foreach(k; keywords)
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1926 idtable[k.str] = k;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	1927 }
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	1928 /+ // Not needed anymore because tokens are stored in a linked list.
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1929 struct State
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1930 {
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1931 Lexer lexer;
209 921d15b1beeb - Added members token and loc to State. aziz parents: 207 diff changeset	1932 Token token;
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1933 char* scanPointer;
209 921d15b1beeb - Added members token and loc to State. aziz parents: 207 diff changeset	1934 int loc;
212 c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1935 string fileName;
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1936 size_t errorLen;
212 c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1937 static State opCall(Lexer lx)
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1938 {
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1939 State s;
212 c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1940 s.lexer = lx;
c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1941 s.token = lx.token;
c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1942 s.scanPointer = lx.p;
c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1943 s.loc = lx.loc;
c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1944 s.fileName = lx.fileName;
c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1945 s.errorLen = lx.errors.length;
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1946 return s;
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1947 }
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1948 void restore()
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1949 {
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1950 lexer.p = scanPointer;
209 921d15b1beeb - Added members token and loc to State. aziz parents: 207 diff changeset	1951 lexer.token = token;
921d15b1beeb - Added members token and loc to State. aziz parents: 207 diff changeset	1952 lexer.loc = loc;
212 c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1953 lexer.fileName = fileName;
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1954 lexer.errors = lexer.errors[0..errorLen];
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1955 }
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1956 }
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1957
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1958 State getState()
df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1959 {
212 c9b9c979a620 - Fix: checking for _Z_. aziz parents: 209 diff changeset	1960 return State(this);
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1961 }
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1962 +/
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	1963
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1964 private void scanNext(ref Token* t)
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	1965 {
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1966 assert(t !is null);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1967 if (t.next)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1968 t = t.next;
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	1969 else if (t != this.tail)
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	1970 {
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1971 Token* new_t = new Token;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1972 scan(*new_t);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1973 new_t.prev = t;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1974 t.next = new_t;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1975 t = new_t;
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	1976 }
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1977 }
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1978
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1979 void peek(ref Token* t)
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1980 {
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1981 scanNext(t);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1982 }
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1983
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1984 TOK nextToken()
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1985 {
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1986 scanNext(this.token);
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1987 return this.token.type;
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	1988 }
511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	1989
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	1990 void error(MID mid, ...)
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1991 {
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	1992 // if (reportErrors)
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	1993 errors ~= new Information(InfoType.Lexer, mid, this.errorLoc, Format(_arguments, _argptr, GetMsg(mid)));
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1994 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1995
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	1996 unittest
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	1997 {
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	1998 Stdout("Testing method Lexer.peek()\n");
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	1999 string sourceText = "unittest { }";
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2000 auto lx = new Lexer(sourceText, null);
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2001
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2002 Token* next = lx.head;
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2003 lx.peek(next);
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2004 assert(next.type == TOK.Unittest);
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2005 lx.peek(next);
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2006 assert(next.type == TOK.LBrace);
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2007 lx.peek(next);
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2008 assert(next.type == TOK.RBrace);
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2009 lx.peek(next);
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2010 assert(next.type == TOK.EOF);
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2011 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2012
306 9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2013 Token* getTokens()
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	2014 {
306 9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2015 while (nextToken() != TOK.EOF)
9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2016 {}
9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2017 return head;
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	2018 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2019
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2020 static bool isNonReservedIdentifier(char[] ident)
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2021 {
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2022 if (ident.length == 0)
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2023 return false;
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2024
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2025 static Identifier[string] reserved_ids_table;
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2026 if (reserved_ids_table is null)
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2027 foreach(k; keywords)
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2028 reserved_ids_table[k.str] = k;
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2029
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2030 size_t idx = 1; // Index to the 2nd character in ident.
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2031 dchar isFirstCharUniAlpha()
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2032 {
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2033 idx = 0;
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2034 // NB: decode() could throw an Exception which would be
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2035 // caught by the next try-catch-block.
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2036 return isUniAlpha(std.utf.decode(ident, idx));
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2037 }
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2038
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2039 try
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2040 {
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2041 if (isidbeg(ident[0]) \|\|
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2042 ident[0] & 128 && isFirstCharUniAlpha())
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2043 {
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2044 foreach (dchar c; ident[idx..$])
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2045 if (!isident(c) && !isUniAlpha(c))
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2046 return false;
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2047 }
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2048 }
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2049 catch (Exception)
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2050 return false;
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2051
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2052 return !(ident in reserved_ids_table);
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2053 }
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2054
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2055 private void encodeUTF8(inout char[] str, dchar d)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2056 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2057 char[6] b;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2058 assert(d > 0x7F, "check for ASCII char before calling encodeUTF8().");
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2059 if (d < 0x800)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2060 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2061 b[0] = 0xC0 \| (d >> 6);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2062 b[1] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2063 str ~= b[0..2];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2064 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2065 else if (d < 0x10000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2066 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2067 b[0] = 0xE0 \| (d >> 12);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2068 b[1] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2069 b[2] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2070 str ~= b[0..3];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2071 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2072 else if (d < 0x200000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2073 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2074 b[0] = 0xF0 \| (d >> 18);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2075 b[1] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2076 b[2] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2077 b[3] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2078 str ~= b[0..4];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2079 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2080 else if (d < 0x4000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2081 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2082 b[0] = 0xF8 \| (d >> 24);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2083 b[1] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2084 b[2] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2085 b[3] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2086 b[4] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2087 str ~= b[0..5];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2088 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2089 else if (d < 0x80000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2090 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2091 b[0] = 0xFC \| (d >> 30);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2092 b[1] = 0x80 \| ((d >> 24) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2093 b[2] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2094 b[3] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2095 b[4] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2096 b[5] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2097 str ~= b[0..6];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2098 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2099 else
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2100 error(MID.InvalidUnicodeCharacter);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2101 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	2102 }
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2103
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2104 unittest
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2105 {
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	2106 Stdout("Testing Lexer.\n");
40 9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2107 string[] toks = [
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2108 ">", ">=", ">>", ">>=", ">>>", ">>>=", "<", "<=", "<>",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2109 "<>=", "<<", "<<=", "!", "!<", "!>", "!<=", "!>=", "!<>",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2110 "!<>=", ".", "..", "...", "&", "&&", "&=", "+", "++",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2111 "+=", "-", "--", "-=", "=", "==", "~", "~=", "*",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2112 "*=", "/", "/=", "^", "^=", "%", "%=", "(", ")",
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2113 "[", "]", "{", "}", ":", ";", "?", ",", "$"
9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2114 ];
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2115
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2116 char[] src;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2117
40 9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2118 foreach (op; toks)
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2119 src ~= op ~ " ";
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2120
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2121 auto lx = new Lexer(src, "");
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2122 auto token = lx.getTokens();
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2123
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2124 uint i;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2125 assert(token == lx.head);
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2126 token = token.next;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2127 do
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2128 {
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2129 assert(i < toks.length);
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	2130 assert(token.srcText == toks[i], Format("Scanned '{0}' but expected '{1}'", token.srcText, toks[i]));
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2131 ++i;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2132 token = token.next;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2133 } while (token.type != TOK.EOF)
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	2134 }
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2135
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2136 unittest
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2137 {
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2138 // Numbers unittest
68 7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2139 // 0L 0ULi 0_L 0_UL 0x0U 0x0p2 0_Fi 0_e2 0_F 0_i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2140 // 0u 0U 0uL 0UL 0L 0LU 0Lu
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2141 // 0Li 0f 0F 0fi 0Fi 0i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2142 // 0b_1_LU 0b1000u
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2143 // 0x232Lu
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2144 }
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2145
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2146 /// ASCII character properties table.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2147 static const int ptable[256] = [
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2148 0, 0, 0, 0, 0, 0, 0, 0, 0,32, 0,32,32, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2149 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2150 32, 0, 0x2200, 0, 0, 0, 0, 0x2700, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2151 7, 7, 7, 7, 7, 7, 7, 7, 6, 6, 0, 0, 0, 0, 0, 0x3f00,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2152 0,12,12,12,12,12,12, 8, 8, 8, 8, 8, 8, 8, 8, 8,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2153 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0, 0x5c00, 0, 0,16,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2154 0, 0x70c, 0x80c,12,12,12, 0xc0c, 8, 8, 8, 8, 8, 8, 8, 0xa08, 8,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2155 8, 8, 0xd08, 8, 0x908, 8, 0xb08, 8, 8, 8, 8, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2156 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2157 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2158 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2159 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2160 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2161 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2162 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2163 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2164 ];
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2165
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2166 enum CProperty
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2167 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2168 Octal = 1,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2169 Digit = 1<<1,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2170 Hex = 1<<2,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2171 Alpha = 1<<3,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2172 Underscore = 1<<4,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2173 Whitespace = 1<<5
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2174 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2175
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2176 const uint EVMask = 0xFF00; // Bit mask for escape value
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2177
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2178 private alias CProperty CP;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2179 int isoctal(char c) { return ptable[c] & CP.Octal; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2180 int isdigit(char c) { return ptable[c] & CP.Digit; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2181 int ishexad(char c) { return ptable[c] & CP.Hex; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2182 int isalpha(char c) { return ptable[c] & CP.Alpha; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2183 int isalnum(char c) { return ptable[c] & (CP.Alpha \| CP.Digit); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2184 int isidbeg(char c) { return ptable[c] & (CP.Alpha \| CP.Underscore); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2185 int isident(char c) { return ptable[c] & (CP.Alpha \| CP.Underscore \| CP.Digit); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2186 int isspace(char c) { return ptable[c] & CP.Whitespace; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2187 int char2ev(char c) { return ptable[c] >> 8; /(ptable[c] & EVMask) >> 8;/ }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2188
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2189 version(gen_ptable)
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2190 static this()
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2191 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2192 alias ptable p;
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2193 assert(p.length == 256);
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2194 // Initialize character properties table.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2195 for (int i; i < p.length; ++i)
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2196 {
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2197 p[i] = 0; // Reset
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2198 if ('0' <= i && i <= '7')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2199 p[i] \|= CP.Octal;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2200 if ('0' <= i && i <= '9')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2201 p[i] \|= CP.Digit;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2202 if (isdigit(i) \|\| 'a' <= i && i <= 'f' \|\| 'A' <= i && i <= 'F')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2203 p[i] \|= CP.Hex;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2204 if ('a' <= i && i <= 'z' \|\| 'A' <= i && i <= 'Z')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2205 p[i] \|= CP.Alpha;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2206 if (i == '_')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2207 p[i] \|= CP.Underscore;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2208 if (i == ' ' \|\| i == '\t' \|\| i == '\v' \|\| i == '\f')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2209 p[i] \|= CP.Whitespace;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2210 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2211 // Store escape sequence values in second byte.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2212 assert(CProperty.max <= ubyte.max, "character property flags and escape value byte overlap.");
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2213 p['\''] \|= 39 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2214 p['"'] \|= 34 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2215 p['?'] \|= 63 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2216 p['\\'] \|= 92 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2217 p['a'] \|= 7 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2218 p['b'] \|= 8 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2219 p['f'] \|= 12 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2220 p['n'] \|= 10 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2221 p['r'] \|= 13 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2222 p['t'] \|= 9 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2223 p['v'] \|= 11 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2224 // Print a formatted array literal.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2225 char[] array = "[\n";
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2226 foreach (i, c; ptable)
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2227 {
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2228 array ~= Format((c>255?" 0x{0:x},":"{0,2},"), c) ~ (((i+1) % 16) ? "":"\n");
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2229 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2230 array[$-2..$] = "\n]";
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2231 Stdout(array).newline;
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2232 }

Mercurial > projects > dil

annotate trunk/src/dil/Lexer.d @ 412:fb31af0fda73