projects/dil: trunk/src/dil/Lexer.d annotate

annotate trunk/src/dil/Lexer.d @ 489:a7291d3ee9d7

Refactored classes that inherit from Node. Added methods addChild(), addOptChild(), addChildren(), addOptChildren() to class Node. Refactored subclasses to use the new methods instead of appending directly to the array Node.children. Moved enums StorageClass and Protection to new module dil.Enums. Added members stc, prot, isStatic() and isPulic() to Declaration.

author	Aziz K?ksal <aziz.koeksal@gmail.com>
date	Mon, 03 Dec 2007 22:44:27 +0100
parents	bccca748d745
children	47be6bfe39cd

rev	line source
0 8ba2570de175 Initial import. aziz parents: diff changeset	1 /++
8ba2570de175 Initial import. aziz parents: diff changeset	2 Author: Aziz Köksal
249 32d354584b28 - Upgraded license notices to GPL3. aziz parents: 239 diff changeset	3 License: GPL3
0 8ba2570de175 Initial import. aziz parents: diff changeset	4 +/
326 4a7359b88c11 - Added package 'dil' to module declarations. aziz parents: 325 diff changeset	5 module dil.Lexer;
327 a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	6 import dil.Token;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	7 import dil.Information;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	8 import dil.Keywords;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	9 import dil.Identifier;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	10 import dil.Messages;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	11 import dil.HtmlEntities;
464 325714d8aa6c Added new module with compiler constants to resolve import cycle. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 434 diff changeset	12 import dil.CompilerInfo;
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	13 import tango.stdc.stdlib : strtof, strtod, strtold;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	14 import tango.stdc.errno : errno, ERANGE;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	15 import tango.stdc.time : time_t, time, ctime;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	16 import tango.stdc.string : strlen;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	17 import std.utf;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	18 import std.uni;
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	19 import common;
0 8ba2570de175 Initial import. aziz parents: diff changeset	20
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	21 const char[3] LS = \u2028; /// Line separator.
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	22 const char[3] PS = \u2029; /// Paragraph separator.
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	23 const dchar LSd = 0x2028;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	24 const dchar PSd = 0x2029;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	25 static assert(LS[0] == PS[0] && LS[1] == PS[1]);
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	26
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	27 /// U+FFFD = �. Used to replace invalid Unicode characters.
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	28 const dchar REPLACEMENT_CHAR = '\uFFFD';
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	29
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	30 const uint _Z_ = 26; /// Control+Z
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	31
0 8ba2570de175 Initial import. aziz parents: diff changeset	32 class Lexer
8ba2570de175 Initial import. aziz parents: diff changeset	33 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	34 Token* head; /// The head of the doubly linked token list.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	35 Token* tail; /// The tail of the linked list. Set in scan().
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	36 Token* token; /// Points to the current token in the token list.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	37 string text; /// The source text.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	38 char[] filePath; /// Path to the source text.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	39 char* p; /// Points to the current character in the source text.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	40 char* end; /// Points one character past the end of the source text.
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	41
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	42 // Members used for error messages:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	43 Information[] errors;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	44 /// Always points to the beginning of the current line.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	45 char* lineBegin;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	46 // Token* newline; /// Current newline token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	47 uint lineNum = 1; /// Current, actual source text line number.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	48 uint lineNum_hline; /// Line number set by #line.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	49 uint inTokenString; /// > 0 if inside q{ }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	50 char[] errorPath; /// The path displayed in error messages.
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	51
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	52 Identifier[string] idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	53
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	54 /++
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	55 Construct a Lexer object.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	56 Params:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	57 text = the UTF-8 source code.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	58 filePath = the path to the source code; used for error messages.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	59 +/
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	60 this(string text, string filePath)
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	61 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	62 this.filePath = this.errorPath = filePath;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	63
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	64 this.text = text;
397 c99f8aeb7b4a Empty source files are handled correctly now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 396 diff changeset	65 if (text.length == 0 \|\| text[$-1] != 0)
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	66 {
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	67 this.text.length = this.text.length + 1;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	68 this.text[$-1] = 0;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	69 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	70
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	71 this.p = this.text.ptr;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	72 this.end = this.p + this.text.length;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	73 this.lineBegin = this.p;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	74 loadKeywords(this.idtable);
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	75
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	76 this.head = new Token;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	77 this.head.type = TOK.HEAD;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	78 this.head.start = this.head.end = this.p;
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	79 this.token = this.head;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	80 // Add a newline as the first token after the head.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	81 auto newline = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	82 newline.type = TOK.Newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	83 newline.start = newline.end = this.p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	84 newline.filePath = this.errorPath;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	85 newline.lineNum = 1;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	86 newline.lineNum_hline = 0;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	87 // Link in.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	88 this.token.next = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	89 newline.prev = this.token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	90 this.token = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	91 // this.newline = newline;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	92 scanShebang();
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	93 }
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	94
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	95 ~this()
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	96 {
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	97 auto token = head.next;
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	98 while (token !is null)
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	99 {
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	100 assert(token.type == TOK.EOF ? token == tail && token.next is null : 1);
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	101 delete token.prev;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	102 token = token.next;
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	103 }
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	104 delete tail;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	105 }
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	106
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	107 /++
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	108 The "shebang" may optionally appear once at the beginning of a file.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	109 Regexp: #![^\EndOfLine]*
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	110 +/
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	111 void scanShebang()
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	112 {
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	113 if (*p == '#' && p[1] == '!')
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	114 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	115 auto t = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	116 t.type = TOK.Shebang;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	117 t.start = p;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	118 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	119 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	120 isascii(*p) \|\| decodeUTF8();
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	121 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	122 this.token.next = t;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	123 t.prev = this.token;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	124 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	125 }
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	126
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	127 void finalizeSpecialToken(ref Token t)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	128 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	129 assert(t.srcText[0..2] == "__");
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	130 switch (t.type)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	131 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	132 case TOK.FILE:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	133 t.str = this.errorPath;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	134 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	135 case TOK.LINE:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	136 t.uint_ = this.errorLineNumber(this.lineNum);
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	137 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	138 case TOK.DATE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	139 TOK.TIME,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	140 TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	141 time_t time_val;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	142 time(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	143 char* str = ctime(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	144 char[] time_str = str[0 .. strlen(str)];
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	145 switch (t.type)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	146 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	147 case TOK.DATE:
346 ce9e2e77743f - Fix: include one space when slicing time_str for __DATE__. aziz parents: 344 diff changeset	148 time_str = time_str[4..11] ~ time_str[20..24] ~ \0; break;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	149 case TOK.TIME:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	150 time_str = time_str[11..19] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	151 case TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	152 time_str = time_str[0..24] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	153 default: assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	154 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	155 t.str = time_str;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	156 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	157 case TOK.VENDOR:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	158 t.str = VENDOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	159 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	160 case TOK.VERSION:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	161 t.uint_ = VERSION_MAJOR*1000 + VERSION_MINOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	162 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	163 default:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	164 assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	165 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	166 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	167
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	168 private void setLineBegin(char* p)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	169 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	170 // Check that we can look behind one character.
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	171 assert((p-1) >= text.ptr && p < end);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	172 // Check that previous character is a newline.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	173 assert(isNewlineEnd(p - 1));
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	174 this.lineBegin = p;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	175 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	176
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	177 private void scanNext(ref Token* t)
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	178 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	179 assert(t !is null);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	180 if (t.next)
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	181 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	182 t = t.next;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	183 // if (t.type == TOK.Newline)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	184 // this.newline = t;
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	185 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	186 else if (t != this.tail)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	187 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	188 Token* new_t = new Token;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	189 scan(*new_t);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	190 new_t.prev = t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	191 t.next = new_t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	192 t = new_t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	193 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	194 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	195
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	196 /// Advance t one token forward.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	197 void peek(ref Token* t)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	198 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	199 scanNext(t);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	200 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	201
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	202 /// Advance to the next token in the source text.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	203 TOK nextToken()
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	204 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	205 scanNext(this.token);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	206 return this.token.type;
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	207 }
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	208
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	209 /// Returns true if d is a Unicode line or paragraph separator.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	210 static bool isUnicodeNewlineChar(dchar d)
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	211 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	212 return d == LSd \|\| d == PSd;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	213 }
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	214
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	215 /// Returns true if p points to a line or paragraph separator.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	216 static bool isUnicodeNewline(char* p)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	217 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	218 return *p == LS[0] && p[1] == LS[1] && (p[2] == LS[2] \|\| p[2] == PS[2]);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	219 }
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	220
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	221 /++
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	222 Returns true if p points to the start of a Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	223 Newline: \n \| \r \| \r\n \| LS \| PS
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	224 +/
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	225 static bool isNewline(char* p)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	226 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	227 return p == '\n' \|\| p == '\r' \|\| isUnicodeNewline(p);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	228 }
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	229
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	230 /// Returns true if p points to the last character of a Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	231 bool isNewlineEnd(char* p)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	232 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	233 if (p == '\n' \|\| p == '\r')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	234 return true;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	235 if (p == LS[2] \|\| p == PS[2])
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	236 if ((p-2) >= text.ptr)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	237 if (p[-1] == LS[1] && p[-2] == LS[0])
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	238 return true;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	239 return false;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	240 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	241
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	242 /++
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	243 Returns true if p points to the first character of an EndOfLine.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	244 EndOfLine: Newline \| 0 \| _Z_
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	245 +/
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	246 static bool isEndOfLine(char* p)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	247 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	248 return isNewline(p) \|\| p == 0 \|\| p == _Z_;
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	249 }
cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	250
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	251 /++
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	252 Scans a Newline and sets p one character past it.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	253 Returns '\n' if scanned or 0 otherwise.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	254 +/
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	255 static dchar scanNewline(ref char* p)
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	256 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	257 switch (*p)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	258 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	259 case '\r':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	260 if (p[1] == '\n')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	261 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	262 case '\n':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	263 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	264 return '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	265 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	266 if (isUnicodeNewline(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	267 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	268 ++p; ++p; ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	269 return '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	270 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	271 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	272 return 0;
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	273 }
cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	274
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	275 /// Returns a Location for the given token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	276 static Location getLocation(Token* token)
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	277 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	278 auto search_t = token.prev;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	279 // Find previous newline token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	280 while (search_t.type != TOK.Newline)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	281 search_t = search_t.prev;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	282 auto filePath = search_t.filePath;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	283 auto lineNum = search_t.lineNum - search_t.lineNum_hline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	284 auto lineBegin = search_t.end;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	285 // Determine actual line begin and line number.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	286 while (1)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	287 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	288 search_t = search_t.next;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	289 if (search_t == token)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	290 break;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	291 // Multiline tokens must be rescanned for newlines.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	292 if (search_t.isMultiline)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	293 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	294 auto p = search_t.start, end = search_t.end;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	295 while (p != end)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	296 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	297 if (Lexer.scanNewline(p) == '\n')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	298 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	299 lineBegin = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	300 ++lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	301 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	302 else
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	303 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	304 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	305 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	306 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	307 return new Location(filePath, lineNum, lineBegin, token.start);
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	308 }
cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	309
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	310 /++
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	311 This is the old scan method.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	312 TODO: profile old and new to see which one is faster.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	313 +/
489 a7291d3ee9d7 Refactored classes that inherit from Node. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	314 public void scan(out Token t)
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	315 in
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	316 {
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	317 assert(text.ptr <= p && p < end);
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	318 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	319 out
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	320 {
397 c99f8aeb7b4a Empty source files are handled correctly now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 396 diff changeset	321 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type));
c99f8aeb7b4a Empty source files are handled correctly now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 396 diff changeset	322 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type));
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	323 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	324 body
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	325 {
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	326 // Scan whitespace.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	327 if (isspace(*p))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	328 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	329 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	330 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	331 {}
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	332 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	333
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	334 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	335 uint c = *p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	336 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	337 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	338 // Newline.
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	339 switch (*p)
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	340 {
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	341 case '\r':
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	342 if (p[1] == '\n')
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	343 ++p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	344 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	345 assert(isNewlineEnd(p));
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	346 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	347 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	348 setLineBegin(p);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	349 // this.newline = &t;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	350 t.type = TOK.Newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	351 t.filePath = this.errorPath;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	352 t.lineNum = lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	353 t.lineNum_hline = lineNum_hline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	354 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	355 return;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	356 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	357 if (isUnicodeNewline(p))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	358 {
412 fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	359 ++p; ++p;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	360 goto case '\n';
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	361 }
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	362 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	363 // Identifier or string literal.
13 e5211758b63c - Added isidbeg() function. aziz parents: 12 diff changeset	364 if (isidbeg(c))
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	365 {
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	366 if (c == 'r' && p[1] == '"' && ++p)
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	367 return scanRawStringLiteral(t);
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	368 if (c == 'x' && p[1] == '"')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	369 return scanHexStringLiteral(t);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	370 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	371 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	372 if (c == 'q' && p[1] == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	373 return scanDelimitedStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	374 if (c == 'q' && p[1] == '{')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	375 return scanTokenStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	376 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	377 // Scan identifier.
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	378 Lidentifier:
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	379 do
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	380 { c = *++p; }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	381 while (isident(c) \|\| !isascii(c) && isUniAlpha(decodeUTF8()))
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	382
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	383 t.end = p;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	384
65 6c21ae79fbb3 - Renamed function Token.span to Token.srcText. aziz parents: 64 diff changeset	385 string str = t.srcText;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	386 Identifier* id = str in idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	387
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	388 if (!id)
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	389 {
327 a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	390 idtable[str] = Identifier(TOK.Identifier, str);
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	391 id = str in idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	392 }
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	393 assert(id);
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	394 t.type = id.type;
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	395 if (t.type == TOK.Identifier)
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	396 return;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	397 if (t.type == TOK.EOF)
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	398 {
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	399 t.type = TOK.EOF;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	400 t.end = p;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	401 tail = &t;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	402 assert(t.srcText == "__EOF__");
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	403 }
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	404 else if (t.isSpecialToken)
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	405 finalizeSpecialToken(t);
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	406 return;
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	407 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	408
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	409 if (isdigit(c))
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	410 return scanNumber(t);
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	411
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	412 if (c == '/')
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	413 {
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	414 c = *++p;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	415 switch(c)
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	416 {
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	417 case '=':
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	418 ++p;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	419 t.type = TOK.DivAssign;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	420 t.end = p;
cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	421 return;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	422 case '+':
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	423 return scanNestedComment(t);
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	424 case '*':
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	425 return scanBlockComment(t);
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	426 case '/':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	427 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	428 isascii(*p) \|\| decodeUTF8();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	429 t.type = TOK.Comment;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	430 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	431 return;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	432 default:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	433 t.type = TOK.Div;
10 3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	434 t.end = p;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	435 return;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	436 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	437 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	438
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	439 switch (c)
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	440 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	441 case '\'':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	442 return scanCharacterLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	443 case '`':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	444 return scanRawStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	445 case '"':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	446 return scanNormalStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	447 case '\\':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	448 char[] buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	449 do
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	450 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	451 c = scanEscapeSequence();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	452 if (isascii(c))
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	453 buffer ~= c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	454 else
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	455 encodeUTF8(buffer, c);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	456 } while (*p == '\\')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	457 buffer ~= 0;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	458 t.type = TOK.String;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	459 t.str = buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	460 t.end = p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	461 return;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	462 case '>': /* > >= >> >>= >>> >>>= */
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	463 c = *++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	464 switch (c)
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	465 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	466 case '=':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	467 t.type = TOK.GreaterEqual;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	468 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	469 case '>':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	470 if (p[1] == '>')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	471 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	472 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	473 if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	474 { ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	475 t.type = TOK.URShiftAssign;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	476 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	477 else
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	478 t.type = TOK.URShift;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	479 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	480 else if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	481 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	482 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	483 t.type = TOK.RShiftAssign;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	484 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	485 else
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	486 t.type = TOK.RShift;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	487 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	488 default:
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	489 t.type = TOK.Greater;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	490 goto Lcommon2;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	491 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	492 assert(0);
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	493 case '<': /* < <= <> <>= << <<= */
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	494 c = *++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	495 switch (c)
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	496 {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	497 case '=':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	498 t.type = TOK.LessEqual;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	499 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	500 case '<':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	501 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	502 ++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	503 t.type = TOK.LShiftAssign;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	504 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	505 else
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	506 t.type = TOK.LShift;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	507 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	508 case '>':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	509 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	510 ++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	511 t.type = TOK.LorEorG;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	512 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	513 else
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	514 t.type = TOK.LorG;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	515 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	516 default:
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	517 t.type = TOK.Less;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	518 goto Lcommon2;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	519 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	520 assert(0);
37 7f3bcb97d017 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 36 diff changeset	521 case '!': /* ! !< !> !<= !>= !<> !<>= */
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	522 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	523 switch (c)
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	524 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	525 case '<':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	526 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	527 if (c == '>')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	528 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	529 if (p[1] == '=') {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	530 ++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	531 t.type = TOK.Unordered;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	532 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	533 else
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	534 t.type = TOK.UorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	535 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	536 else if (c == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	537 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	538 t.type = TOK.UorG;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	539 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	540 else {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	541 t.type = TOK.UorGorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	542 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	543 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	544 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	545 case '>':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	546 if (p[1] == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	547 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	548 ++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	549 t.type = TOK.UorL;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	550 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	551 else
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	552 t.type = TOK.UorLorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	553 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	554 case '=':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	555 t.type = TOK.NotEqual;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	556 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	557 default:
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	558 t.type = TOK.Not;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	559 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	560 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	561 assert(0);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	562 case '.': /* . .[0-9] .. ... */
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	563 if (p[1] == '.')
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	564 {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	565 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	566 if (p[1] == '.') {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	567 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	568 t.type = TOK.Ellipses;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	569 }
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	570 else
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	571 t.type = TOK.Slice;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	572 }
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	573 else if (isdigit(p[1]))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	574 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	575 return scanReal(t);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	576 }
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	577 else
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	578 t.type = TOK.Dot;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	579 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	580 case '\|': /* \| \|\| \|= */
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	581 c = *++p;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	582 if (c == '=')
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	583 t.type = TOK.OrAssign;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	584 else if (c == '\|')
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	585 t.type = TOK.OrLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	586 else {
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	587 t.type = TOK.OrBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	588 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	589 }
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	590 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	591 case '&': /* & && &= */
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	592 c = *++p;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	593 if (c == '=')
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	594 t.type = TOK.AndAssign;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	595 else if (c == '&')
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	596 t.type = TOK.AndLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	597 else {
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	598 t.type = TOK.AndBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	599 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	600 }
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	601 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	602 case '+': /* + ++ += */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	603 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	604 if (c == '=')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	605 t.type = TOK.PlusAssign;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	606 else if (c == '+')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	607 t.type = TOK.PlusPlus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	608 else {
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	609 t.type = TOK.Plus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	610 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	611 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	612 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	613 case '-': /* - -- -= */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	614 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	615 if (c == '=')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	616 t.type = TOK.MinusAssign;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	617 else if (c == '-')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	618 t.type = TOK.MinusMinus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	619 else {
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	620 t.type = TOK.Minus;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	621 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	622 }
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	623 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	624 case '=': /* = == */
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	625 if (p[1] == '=') {
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	626 ++p;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	627 t.type = TOK.Equal;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	628 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	629 else
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	630 t.type = TOK.Assign;
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	631 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	632 case '~': /* ~ ~= */
27 43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	633 if (p[1] == '=') {
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	634 ++p;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	635 t.type = TOK.CatAssign;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	636 }
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	637 else
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	638 t.type = TOK.Tilde;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	639 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	640 case '': / * = /
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	641 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	642 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	643 t.type = TOK.MulAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	644 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	645 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	646 t.type = TOK.Mul;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	647 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	648 case '^': /* ^ ^= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	649 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	650 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	651 t.type = TOK.XorAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	652 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	653 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	654 t.type = TOK.Xor;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	655 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	656 case '%': /* % %= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	657 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	658 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	659 t.type = TOK.ModAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	660 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	661 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	662 t.type = TOK.Mod;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	663 goto Lcommon;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	664 // Single character tokens:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	665 case '(':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	666 t.type = TOK.LParen;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	667 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	668 case ')':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	669 t.type = TOK.RParen;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	670 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	671 case '[':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	672 t.type = TOK.LBracket;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	673 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	674 case ']':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	675 t.type = TOK.RBracket;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	676 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	677 case '{':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	678 t.type = TOK.LBrace;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	679 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	680 case '}':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	681 t.type = TOK.RBrace;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	682 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	683 case ':':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	684 t.type = TOK.Colon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	685 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	686 case ';':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	687 t.type = TOK.Semicolon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	688 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	689 case '?':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	690 t.type = TOK.Question;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	691 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	692 case ',':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	693 t.type = TOK.Comma;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	694 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	695 case '$':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	696 t.type = TOK.Dollar;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	697 Lcommon:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	698 ++p;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	699 Lcommon2:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	700 t.end = p;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	701 return;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	702 case '#':
360 b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence(). aziz parents: 350 diff changeset	703 return scanSpecialTokenSequence(t);
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	704 default:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	705 }
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	706
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	707 // Check for EOF
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	708 if (c == 0 \|\| c == _Z_)
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	709 {
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	710 assert(p == 0 \|\| p == _Z_);
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	711 t.type = TOK.EOF;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	712 t.end = p;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	713 tail = &t;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	714 assert(t.start == t.end);
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	715 return;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	716 }
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	717
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	718 if (!isascii(c))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	719 {
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	720 c = decodeUTF8();
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	721 if (isUniAlpha(c))
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	722 goto Lidentifier;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	723 }
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	724
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	725 error(t.start, MID.IllegalCharacter, cast(dchar)c);
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	726
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	727 ++p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	728 t.type = TOK.Illegal;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	729 t.dchar_ = c;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	730 t.end = p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	731 return;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	732 }
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	733 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	734
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	735 template toUint(char[] T)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	736 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	737 static assert(0 < T.length && T.length <= 4);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	738 static if (T.length == 1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	739 const uint toUint = T[0];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	740 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	741 const uint toUint = (T[0] << ((T.length-1)*8)) \| toUint!(T[1..$]);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	742 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	743 static assert(toUint!("\xAA\xBB\xCC\xDD") == 0xAABBCCDD);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	744
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	745 // Can't use this yet due to a bug in DMD (bug id=1534).
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	746 template case_(char[] str, TOK tok, char[] label)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	747 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	748 const char[] case_ =
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	749 `case `~toUint!(str).stringof~`:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	750
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	751 goto `~label~`;`;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	752 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	753
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	754 template case_L4(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	755 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	756 const char[] case_L4 = case_!(str, tok, "Lcommon_4");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	757 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	758
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	759 template case_L3(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	760 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	761 const char[] case_L3 = case_!(str, tok, "Lcommon_3");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	762 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	763
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	764 template case_L2(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	765 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	766 const char[] case_L2 = case_!(str, tok, "Lcommon_2");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	767 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	768
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	769 template case_L1(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	770 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	771 const char[] case_L3 = case_!(str, tok, "Lcommon");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	772 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	773
489 a7291d3ee9d7 Refactored classes that inherit from Node. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	774 public void scan_(out Token t)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	775 in
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	776 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	777 assert(text.ptr <= p && p < end);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	778 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	779 out
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	780 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	781 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type));
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	782 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type));
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	783 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	784 body
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	785 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	786 // Scan whitespace.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	787 if (isspace(*p))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	788 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	789 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	790 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	791 {}
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	792 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	793
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	794 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	795 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	796 // Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	797 switch (*p)
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	798 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	799 case '\r':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	800 if (p[1] == '\n')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	801 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	802 case '\n':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	803 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	804 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	805 ++lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	806 setLineBegin(p);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	807 // this.newline = &t;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	808 t.type = TOK.Newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	809 t.filePath = this.errorPath;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	810 t.lineNum = lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	811 t.lineNum_hline = lineNum_hline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	812 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	813 return;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	814 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	815 if (isUnicodeNewline(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	816 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	817 ++p; ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	818 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	819 }
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	820 }
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	821
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	822 uint c = *p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	823 assert(end - p != 0);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	824 switch (end - p)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	825 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	826 case 1:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	827 goto L1character;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	828 case 2:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	829 c <<= 8; c \|= p[1];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	830 goto L2characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	831 case 3:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	832 c <<= 8; c \|= p[1]; c <<= 8; c \|= p[2];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	833 goto L3characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	834 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	835 version(BigEndian)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	836 c = cast(uint)p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	837 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	838 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	839 c <<= 8; c \|= p[1]; c <<= 8; c \|= p[2]; c <<= 8; c \|= p[3];
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	840 /+
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	841 c = cast(uint)p;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	842 asm
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	843 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	844 mov EDX, c;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	845 bswap EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	846 mov c, EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	847 }
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	848 +/
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	849 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	850 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	851
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	852 // 4 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	853 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	854 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	855 case toUint!(">>>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	856 t.type = TOK.RShiftAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	857 goto Lcommon_4;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	858 case toUint!("!<>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	859 t.type = TOK.Unordered;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	860 Lcommon_4:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	861 p += 4;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	862 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	863 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	864 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	865 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	866
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	867 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	868 L3characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	869 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	870 // 3 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	871 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	872 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	873 case toUint!(">>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	874 t.type = TOK.RShiftAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	875 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	876 case toUint!(">>>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	877 t.type = TOK.URShift;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	878 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	879 case toUint!("<>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	880 t.type = TOK.LorEorG;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	881 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	882 case toUint!("<<="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	883 t.type = TOK.LShiftAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	884 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	885 case toUint!("!<="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	886 t.type = TOK.UorG;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	887 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	888 case toUint!("!>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	889 t.type = TOK.UorL;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	890 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	891 case toUint!("!<>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	892 t.type = TOK.UorE;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	893 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	894 case toUint!("..."):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	895 t.type = TOK.Ellipses;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	896 Lcommon_3:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	897 p += 3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	898 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	899 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	900 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	901 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	902
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	903 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	904 L2characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	905 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	906 // 2 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	907 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	908 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	909 case toUint!("/+"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	910 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	911 return scanNestedComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	912 case toUint!("/*"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	913 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	914 return scanBlockComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	915 case toUint!("//"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	916 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	917 assert(*p == '/');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	918 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	919 isascii(*p) \|\| decodeUTF8();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	920 t.type = TOK.Comment;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	921 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	922 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	923 case toUint!(">="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	924 t.type = TOK.GreaterEqual;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	925 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	926 case toUint!(">>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	927 t.type = TOK.RShift;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	928 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	929 case toUint!("<<"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	930 t.type = TOK.LShift;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	931 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	932 case toUint!("<="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	933 t.type = TOK.LessEqual;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	934 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	935 case toUint!("<>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	936 t.type = TOK.LorG;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	937 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	938 case toUint!("!<"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	939 t.type = TOK.UorGorE;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	940 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	941 case toUint!("!>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	942 t.type = TOK.UorLorE;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	943 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	944 case toUint!("!="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	945 t.type = TOK.NotEqual;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	946 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	947 case toUint!(".."):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	948 t.type = TOK.Slice;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	949 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	950 case toUint!("&&"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	951 t.type = TOK.AndLogical;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	952 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	953 case toUint!("&="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	954 t.type = TOK.AndAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	955 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	956 case toUint!("\|\|"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	957 t.type = TOK.OrLogical;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	958 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	959 case toUint!("\|="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	960 t.type = TOK.OrAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	961 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	962 case toUint!("++"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	963 t.type = TOK.PlusPlus;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	964 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	965 case toUint!("+="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	966 t.type = TOK.PlusAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	967 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	968 case toUint!("--"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	969 t.type = TOK.MinusMinus;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	970 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	971 case toUint!("-="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	972 t.type = TOK.MinusAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	973 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	974 case toUint!("=="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	975 t.type = TOK.Equal;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	976 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	977 case toUint!("~="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	978 t.type = TOK.CatAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	979 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	980 case toUint!("*="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	981 t.type = TOK.MulAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	982 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	983 case toUint!("/="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	984 t.type = TOK.DivAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	985 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	986 case toUint!("^="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	987 t.type = TOK.XorAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	988 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	989 case toUint!("%="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	990 t.type = TOK.ModAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	991 Lcommon_2:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	992 p += 2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	993 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	994 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	995 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	996 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	997
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	998 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	999 L1character:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1000 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1001 assert(p == c, Format("p={0},c={1}", p, cast(dchar)c));
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1002 // 1 character tokens.
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	1003 // TODO: consider storing the token type in ptable.
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1004 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1005 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1006 case '\'':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1007 return scanCharacterLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1008 case '`':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1009 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1010 case '"':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1011 return scanNormalStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1012 case '\\':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1013 char[] buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1014 do
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1015 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1016 c = scanEscapeSequence();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1017 if (isascii(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1018 buffer ~= c;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1019 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1020 encodeUTF8(buffer, c);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1021 } while (*p == '\\')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1022 buffer ~= 0;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1023 t.type = TOK.String;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1024 t.str = buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1025 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1026 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1027 case '<':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1028 t.type = TOK.Greater;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1029 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1030 case '>':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1031 t.type = TOK.Less;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1032 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1033 case '^':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1034 t.type = TOK.Xor;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1035 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1036 case '!':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1037 t.type = TOK.Not;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1038 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1039 case '.':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1040 if (isdigit(p[1]))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1041 return scanReal(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1042 t.type = TOK.Dot;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1043 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1044 case '&':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1045 t.type = TOK.AndBinary;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1046 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1047 case '\|':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1048 t.type = TOK.OrBinary;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1049 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1050 case '+':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1051 t.type = TOK.Plus;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1052 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1053 case '-':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1054 t.type = TOK.Minus;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1055 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1056 case '=':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1057 t.type = TOK.Assign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1058 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1059 case '~':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1060 t.type = TOK.Tilde;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1061 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1062 case '*':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1063 t.type = TOK.Mul;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1064 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1065 case '/':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1066 t.type = TOK.Div;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1067 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1068 case '%':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1069 t.type = TOK.Mod;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1070 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1071 case '(':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1072 t.type = TOK.LParen;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1073 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1074 case ')':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1075 t.type = TOK.RParen;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1076 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1077 case '[':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1078 t.type = TOK.LBracket;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1079 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1080 case ']':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1081 t.type = TOK.RBracket;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1082 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1083 case '{':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1084 t.type = TOK.LBrace;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1085 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1086 case '}':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1087 t.type = TOK.RBrace;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1088 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1089 case ':':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1090 t.type = TOK.Colon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1091 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1092 case ';':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1093 t.type = TOK.Semicolon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1094 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1095 case '?':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1096 t.type = TOK.Question;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1097 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1098 case ',':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1099 t.type = TOK.Comma;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1100 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1101 case '$':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1102 t.type = TOK.Dollar;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1103 Lcommon:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1104 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1105 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1106 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1107 case '#':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1108 return scanSpecialTokenSequence(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1109 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1110 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1111
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1112 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1113 assert(*p == c);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1114
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1115 // TODO: consider moving isidbeg() and isdigit() up.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1116 if (isidbeg(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1117 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1118 if (c == 'r' && p[1] == '"' && ++p)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1119 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1120 if (c == 'x' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1121 return scanHexStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1122 version(D2)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1123 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1124 if (c == 'q' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1125 return scanDelimitedStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1126 if (c == 'q' && p[1] == '{')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1127 return scanTokenStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1128 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1129 // Scan identifier.
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1130 Lidentifier:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1131 do
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1132 { c = *++p; }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1133 while (isident(c) \|\| !isascii(c) && isUniAlpha(decodeUTF8()))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1134
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1135 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1136
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1137 string str = t.srcText;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1138 Identifier* id = str in idtable;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1139
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1140 if (!id)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1141 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1142 idtable[str] = Identifier(TOK.Identifier, str);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1143 id = str in idtable;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1144 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1145 assert(id);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1146 t.type = id.type;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1147 if (t.type == TOK.Identifier)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1148 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1149 if (t.type == TOK.EOF)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1150 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1151 t.type = TOK.EOF;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1152 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1153 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1154 assert(t.srcText == "__EOF__");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1155 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1156 else if (t.isSpecialToken)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1157 finalizeSpecialToken(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1158 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1159 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1160
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1161 if (isdigit(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1162 return scanNumber(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1163
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1164 // Check for EOF
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1165 if (c == 0 \|\| c == _Z_)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1166 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1167 assert(p == 0 \|\| p == _Z_, *p~"");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1168 t.type = TOK.EOF;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1169 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1170 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1171 assert(t.start == t.end);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1172 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1173 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1174
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1175 if (!isascii(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1176 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1177 c = decodeUTF8();
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1178 if (isUniAlpha(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1179 goto Lidentifier;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1180 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1181
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1182 error(t.start, MID.IllegalCharacter, cast(dchar)c);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1183
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1184 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1185 t.type = TOK.Illegal;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1186 t.dchar_ = c;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1187 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1188 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1189 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1190
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1191 void scanBlockComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1192 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1193 assert(p[-1] == '/' && p == '');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1194 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1195 auto tokenLineBegin = lineBegin;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1196 uint c;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1197 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1198 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1199 c = *++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1200 LswitchBC: // only jumped to from default case of next switch(c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1201 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1202 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1203 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1204 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1205 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1206 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1207 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1208 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1209 setLineBegin(p+1);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1210 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1211 case 0, _Z_:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1212 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedBlockComment);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1213 goto LreturnBC;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1214 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1215 if (!isascii(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1216 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1217 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1218 if (isUnicodeNewlineChar(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1219 goto case '\n';
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1220 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1221 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1222 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1223
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1224 c <<= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1225 c \|= *++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1226 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1227 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1228 case toUint!("*/"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1229 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1230 LreturnBC:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1231 t.type = TOK.Comment;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1232 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1233 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1234 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1235 c &= char.max;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1236 goto LswitchBC;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1237 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1238 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1239 assert(0);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1240 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1241
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1242 void scanNestedComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1243 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1244 assert(p[-1] == '/' && *p == '+');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1245 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1246 auto tokenLineBegin = lineBegin;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1247 uint level = 1;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1248 uint c;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1249 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1250 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1251 c = *++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1252 LswitchNC: // only jumped to from default case of next switch(c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1253 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1254 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1255 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1256 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1257 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1258 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1259 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1260 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1261 setLineBegin(p+1);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1262 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1263 case 0, _Z_:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1264 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedNestedComment);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1265 goto LreturnNC;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1266 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1267 if (!isascii(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1268 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1269 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1270 if (isUnicodeNewlineChar(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1271 goto case '\n';
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1272 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1273 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1274 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1275
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1276 c <<= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1277 c \|= *++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1278 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1279 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1280 case toUint!("/+"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1281 ++level;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1282 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1283 case toUint!("+/"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1284 if (--level == 0)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1285 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1286 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1287 LreturnNC:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1288 t.type = TOK.Comment;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1289 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1290 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1291 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1292 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1293 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1294 c &= char.max;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1295 goto LswitchNC;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1296 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1297 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1298 assert(0);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1299 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1300
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1301 char scanPostfix()
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1302 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1303 assert(p[-1] == '"' \|\| p[-1] == '`' \|\|
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1304 { version(D2) return p[-1] == '}';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1305 else return 0; }()
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1306 );
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1307 switch (*p)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1308 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1309 case 'c':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1310 case 'w':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1311 case 'd':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1312 return *p++;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1313 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1314 return 0;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1315 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1316 assert(0);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1317 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1318
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1319 void scanNormalStringLiteral(ref Token t)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1320 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1321 assert(*p == '"');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1322 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1323 auto tokenLineBegin = lineBegin;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1324 t.type = TOK.String;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1325 char[] buffer;
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1326 uint c;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1327 while (1)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1328 {
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1329 c = *++p;
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1330 switch (c)
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1331 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1332 case '"':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1333 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1334 t.pf = scanPostfix();
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1335 Lreturn:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1336 t.str = buffer ~ '\0';
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1337 t.end = p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1338 return;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1339 case '\\':
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1340 c = scanEscapeSequence();
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1341 --p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1342 if (isascii(c))
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1343 break;
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1344 encodeUTF8(buffer, c);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1345 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1346 case '\r':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1347 if (p[1] == '\n')
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1348 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1349 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1350 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1351 c = '\n'; // Convert Newline to \n.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1352 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1353 setLineBegin(p+1);
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1354 break;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1355 case 0, _Z_:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1356 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedString);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1357 goto Lreturn;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1358 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1359 if (!isascii(c))
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1360 {
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1361 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1362 if (isUnicodeNewlineChar(c))
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1363 goto case '\n';
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1364 encodeUTF8(buffer, c);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1365 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1366 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1367 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1368 assert(isascii(c));
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1369 buffer ~= c;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1370 }
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1371 assert(0);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1372 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1373
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1374 void scanCharacterLiteral(ref Token t)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1375 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1376 assert(*p == '\'');
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1377 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1378 t.type = TOK.CharLiteral;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1379 switch (*p)
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1380 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1381 case '\\':
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1382 switch (p[1])
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1383 {
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1384 case 'u':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1385 t.type = TOK.WCharLiteral; break;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1386 case 'U':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1387 t.type = TOK.DCharLiteral; break;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1388 default:
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1389 }
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1390 t.dchar_ = scanEscapeSequence();
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1391 break;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1392 case '\'':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1393 error(t.start, MID.EmptyCharacterLiteral);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1394 break;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1395 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1396 if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1397 break;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1398 uint c = *p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1399 if (!isascii(c))
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1400 {
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1401 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1402 t.type = c <= 0xFFFF ? TOK.WCharLiteral : TOK.DCharLiteral;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1403 }
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1404 t.dchar_ = c;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1405 ++p;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1406 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1407
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1408 if (*p == '\'')
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1409 ++p;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1410 else
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1411 error(t.start, MID.UnterminatedCharacterLiteral);
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1412 t.end = p;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1413 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1414
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1415 void scanRawStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1416 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1417 assert(p == '`' \|\| p == '"' && p[-1] == 'r');
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1418 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1419 auto tokenLineBegin = lineBegin;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1420 t.type = TOK.String;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1421 uint delim = *p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1422 char[] buffer;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1423 uint c;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1424 while (1)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1425 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1426 c = *++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1427 switch (c)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1428 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1429 case '\r':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1430 if (p[1] == '\n')
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1431 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1432 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1433 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1434 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1435 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1436 setLineBegin(p+1);
52 f65a83c27638 - Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well. aziz parents: 51 diff changeset	1437 break;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1438 case '`':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1439 case '"':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1440 if (c == delim)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1441 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1442 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1443 t.pf = scanPostfix();
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1444 Lreturn:
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1445 t.str = buffer ~ '\0';
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1446 t.end = p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1447 return;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1448 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1449 break;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1450 case 0, _Z_:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1451 error(tokenLineNum, tokenLineBegin, t.start,
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1452 delim == 'r' ? MID.UnterminatedRawString : MID.UnterminatedBackQuoteString);
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1453 goto Lreturn;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1454 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1455 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1456 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1457 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1458 if (isUnicodeNewlineChar(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1459 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1460 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1461 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1462 }
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1463 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1464 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1465 buffer ~= c;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1466 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1467 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1468 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1469
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1470 void scanHexStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1471 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1472 assert(p[0] == 'x' && p[1] == '"');
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1473 t.type = TOK.String;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1474
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1475 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1476 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1477
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1478 uint c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1479 ubyte[] buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1480 ubyte h; // hex number
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1481 uint n; // number of hex digits
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1482
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1483 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1484 assert(*p == '"');
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1485 while (1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1486 {
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1487 c = *++p;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1488 switch (c)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1489 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1490 case '"':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1491 if (n & 1)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1492 error(tokenLineNum, tokenLineBegin, t.start, MID.OddNumberOfDigitsInHexString);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1493 ++p;
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1494 t.pf = scanPostfix();
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1495 Lreturn:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1496 t.str = cast(string) (buffer ~= 0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1497 t.end = p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1498 return;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1499 case '\r':
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1500 if (p[1] == '\n')
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1501 ++p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1502 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1503 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1504 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1505 setLineBegin(p+1);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1506 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1507 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1508 if (ishexad(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1509 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1510 if (c <= '9')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1511 c -= '0';
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1512 else if (c <= 'F')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1513 c -= 'A' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1514 else
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1515 c -= 'a' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1516
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1517 if (n & 1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1518 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1519 h <<= 4;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1520 h \|= c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1521 buffer ~= h;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1522 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1523 else
112 004d98df65af - Implemented parseInterfaceDeclaration(). aziz parents: 105 diff changeset	1524 h = cast(ubyte)c;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1525 ++n;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1526 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1527 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1528 else if (isspace(c))
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1529 continue; // Skip spaces.
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1530 else if (c == 0 \|\| c == _Z_)
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1531 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1532 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedHexString);
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1533 t.pf = 0;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1534 goto Lreturn;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1535 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1536 else
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1537 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1538 auto errorAt = p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1539 if (!isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1540 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1541 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1542 if (isUnicodeNewlineChar(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1543 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1544 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1545 error(errorAt, MID.NonHexCharInHexString, cast(dchar)c);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1546 }
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1547 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1548 }
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1549 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1550 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1551
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1552 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1553 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1554 void scanDelimitedStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1555 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1556 assert(p[0] == 'q' && p[1] == '"');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1557 t.type = TOK.String;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1558
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1559 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1560 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1561
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1562 char[] buffer;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1563 dchar opening_delim = 0, // 0 if no nested delimiter or '[', '(', '<', '{'
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1564 closing_delim; // Will be ']', ')', '>', '},
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1565 // the first character of an identifier or
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1566 // any other Unicode/ASCII character.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1567 char[] str_delim; // Identifier delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1568 uint level = 1; // Counter for nestable delimiters.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1569
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1570 ++p; ++p; // Skip q"
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1571 uint c = *p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1572 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1573 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1574 case '(':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1575 opening_delim = c;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1576 closing_delim = ')'; // c + 1
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1577 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1578 case '[', '<', '{':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1579 opening_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1580 closing_delim = c + 2; // Get to closing counterpart. Feature of ASCII table.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1581 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1582 default:
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1583 dchar scanNewline()
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1584 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1585 switch (*p)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1586 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1587 case '\r':
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1588 if (p[1] == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1589 ++p;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1590 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1591 assert(isNewlineEnd(p));
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1592 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1593 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1594 setLineBegin(p);
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1595 return '\n';
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1596 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1597 if (isUnicodeNewline(p))
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1598 {
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1599 ++p; ++p;
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1600 goto case '\n';
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1601 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1602 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1603 return 0;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1604 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1605 // Skip leading newlines:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1606 while (scanNewline() != 0)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1607 {}
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1608 assert(!isNewline(p));
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1609
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1610 char* begin = p;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1611 c = *p;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1612 closing_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1613 // TODO: Check for non-printable characters?
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1614 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1615 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1616 closing_delim = decodeUTF8();
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1617 if (!isUniAlpha(closing_delim))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1618 break; // Not an identifier.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1619 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1620 else if (!isidbeg(c))
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1621 break; // Not an identifier.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1622
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1623 // Parse Identifier + EndOfLine
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1624 do
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1625 { c = *++p; }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1626 while (isident(c) \|\| !isascii(c) && isUniAlpha(decodeUTF8()))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1627 // Store identifier
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1628 str_delim = begin[0..p-begin];
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1629 // Scan newline
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1630 if (scanNewline() == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1631 --p; // Go back one because of "c = *++p;" in main loop.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1632 else
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1633 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1634 // TODO: error(p, MID.ExpectedNewlineAfterIdentDelim);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1635 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1636 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1637
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1638 bool checkStringDelim(char* p)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1639 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1640 assert(str_delim.length != 0);
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1641 if (buffer[$-1] == '\n' && // Last character copied to buffer must be '\n'.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1642 end-p >= str_delim.length && // Check remaining length.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1643 p[0..str_delim.length] == str_delim) // Compare.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1644 return true;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1645 return false;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1646 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1647
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1648 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1649 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1650 c = *++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1651 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1652 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1653 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1654 if (p[1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1655 ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1656 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1657 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1658 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1659 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1660 setLineBegin(p+1);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1661 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1662 case 0, _Z_:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1663 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedDelimitedString);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1664 goto Lreturn3;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1665 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1666 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1667 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1668 auto begin = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1669 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1670 if (isUnicodeNewlineChar(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1671 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1672 if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1673 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1674 if (str_delim.length)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1675 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1676 if (checkStringDelim(begin))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1677 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1678 p = begin + str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1679 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1680 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1681 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1682 else
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1683 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1684 assert(level == 1);
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1685 --level;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1686 goto Lreturn;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1687 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1688 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1689 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1690 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1691 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1692 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1693 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1694 if (c == opening_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1695 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1696 else if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1697 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1698 if (str_delim.length)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1699 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1700 if (checkStringDelim(p))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1701 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1702 p += str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1703 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1704 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1705 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1706 else if (--level == 0)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1707 goto Lreturn;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1708 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1709 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1710 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1711 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1712 buffer ~= c;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1713 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1714 Lreturn: // Character delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1715 assert(c == closing_delim);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1716 assert(level == 0);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1717 ++p; // Skip closing delimiter.
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1718 Lreturn2: // String delimiter.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1719 if (*p == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1720 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1721 else
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1722 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1723 // TODO: error(p, MID.ExpectedDblQuoteAfterDelim, str_delim.length ? str_delim : closing_delim~"");
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1724 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1725
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1726 t.pf = scanPostfix();
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1727 Lreturn3: // Error.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1728 t.str = buffer ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1729 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1730 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1731
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1732 void scanTokenStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1733 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1734 assert(p[0] == 'q' && p[1] == '{');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1735 t.type = TOK.String;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1736
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1737 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1738 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1739
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1740 // A guard against changes to particular members:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1741 // this.lineNum_hline and this.errorPath
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1742 ++inTokenString;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1743
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1744 uint lineNum = this.lineNum;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1745 uint level = 1;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1746
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1747 ++p; ++p; // Skip q{
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1748
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1749 auto prev_t = &t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1750 Token* token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1751 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1752 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1753 token = new Token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1754 scan(*token);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1755 // Save the tokens in a doubly linked list.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1756 // Could be useful for various tools.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1757 token.prev = prev_t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1758 prev_t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1759 prev_t = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1760 switch (token.type)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1761 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1762 case TOK.LBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1763 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1764 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1765 case TOK.RBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1766 if (--level == 0)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1767 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1768 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1769 t.next = null;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1770 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1771 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1772 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1773 case TOK.EOF:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1774 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedTokenString);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1775 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1776 t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1777 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1778 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1779 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1780 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1781 break; // Exit loop.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1782 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1783
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1784 assert(token.type == TOK.RBrace \|\| token.type == TOK.EOF);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1785 assert(token.type == TOK.RBrace && t.next is null \|\|
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1786 token.type == TOK.EOF && t.next !is null);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1787
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1788 char[] buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1789 // token points to } or EOF
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1790 if (token.type == TOK.EOF)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1791 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1792 t.end = token.start;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1793 buffer = t.srcText[2..$].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1794 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1795 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1796 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1797 // Assign to buffer before scanPostfix().
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1798 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1799 buffer = t.srcText[2..$-1].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1800 t.pf = scanPostfix();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1801 t.end = p; // Assign again because of postfix.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1802 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1803 // Convert newlines to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1804 if (lineNum != this.lineNum)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1805 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1806 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1807 uint i, j;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1808 for (; i < buffer.length; ++i)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1809 switch (buffer[i])
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1810 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1811 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1812 if (buffer[i+1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1813 ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1814 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1815 assert(isNewlineEnd(buffer.ptr + i));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1816 buffer[j++] = '\n'; // Convert Newline to '\n'.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1817 break;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1818 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1819 if (isUnicodeNewline(buffer.ptr + i))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1820 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1821 ++i; ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1822 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1823 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1824 buffer[j++] = buffer[i]; // Copy.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1825 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1826 buffer.length = j; // Adjust length.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1827 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1828 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1829 t.str = buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1830
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1831 --inTokenString;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1832 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1833 } // version(D2)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1834
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1835 dchar scanEscapeSequence()
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1836 out(result)
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1837 { assert(isEncodable(result)); }
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1838 body
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1839 {
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1840 assert(*p == '\\');
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1841
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1842 auto sequenceStart = p; // Used for error reporting.
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1843
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1844 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1845 uint c = char2ev(*p);
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1846 if (c)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1847 {
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1848 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1849 return c;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1850 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1851
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1852 uint digits = 2;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1853
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1854 switch (*p)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1855 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1856 case 'x':
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1857 assert(c == 0);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1858 while (1)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1859 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1860 ++p;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1861 if (ishexad(*p))
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1862 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1863 c *= 16;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1864 if (*p <= '9')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1865 c += *p - '0';
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1866 else if (*p <= 'F')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1867 c += *p - 'A' + 10;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1868 else
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1869 c += *p - 'a' + 10;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1870
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1871 if (!--digits)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1872 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1873 ++p;
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1874 if (isEncodable(c))
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1875 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1876
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1877 error(sequenceStart, MID.InvalidUnicodeEscapeSequence, sequenceStart[0..p-sequenceStart]);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1878 break;
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1879 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1880 continue;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1881 }
432 3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 430 diff changeset	1882
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1883 error(sequenceStart, MID.InsufficientHexDigits);
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1884 break;
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1885 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1886 break;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1887 case 'u':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1888 digits = 4;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1889 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1890 case 'U':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1891 digits = 8;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1892 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1893 default:
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1894 if (isoctal(*p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1895 {
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1896 assert(c == 0);
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1897 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1898 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1899 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1900 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1901 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1902 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1903 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1904 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1905 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1906 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1907 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1908 ++p;
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1909 return c; // Return valid escape value.
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1910 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1911 else if(*p == '&')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1912 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1913 if (isalpha(*++p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1914 {
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1915 auto begin = p;
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1916 while (isalnum(*++p))
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1917 {}
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1918
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1919 if (*p == ';')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1920 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1921 // Pass entity excluding '&' and ';'.
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1922 c = entity2Unicode(begin[0..p - begin]);
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1923 ++p; // Skip ;
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1924 if (c != 0xFFFF)
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1925 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1926 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1927 error(sequenceStart, MID.UndefinedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1928 }
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1929 else
432 3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 430 diff changeset	1930 error(sequenceStart, MID.UnterminatedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1931 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1932 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1933 error(sequenceStart, MID.InvalidBeginHTMLEntity);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1934 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1935 else if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1936 error(sequenceStart, MID.UndefinedEscapeSequence,
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1937 (p == 0 \|\| p == _Z_) ? `\EOF` : `\NewLine`);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1938 else
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1939 {
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1940 char[] str = `\`;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1941 if (isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1942 str ~= *p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1943 else
387 ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1944 encodeUTF8(str, decodeUTF8());
ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1945 ++p;
ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1946 // TODO: check for unprintable character?
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1947 error(sequenceStart, MID.UndefinedEscapeSequence, str);
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1948 }
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1949 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1950 return REPLACEMENT_CHAR; // Error: return replacement character.
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1951 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1952
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1953 /*
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1954 IntegerLiteral:= (Dec\|Hex\|Bin\|Oct)Suffix?
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1955 Dec:= (0\|[1-9][0-9_]*)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1956 Hex:= 0[xX] HexDigits
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1957 Bin:= 0[bB][01_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1958 Oct:= 0[0-7_]+
68 7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	1959 Suffix:= (L[uU]?\|[uU]L?)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1960 HexDigits:= [0-9a-zA-Z_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1961
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1962 Invalid: "0b_", "0x_", "._"
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1963 */
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1964 void scanNumber(ref Token t)
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1965 {
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1966 ulong ulong_;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1967 bool overflow;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1968 bool isDecimal;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1969 size_t digits;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1970
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1971 if (*p != '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1972 goto LscanInteger;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1973 ++p; // skip zero
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1974 // check for xX bB ...
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1975 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1976 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1977 case 'x','X':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1978 goto LscanHex;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1979 case 'b','B':
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1980 goto LscanBinary;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1981 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1982 if (p[1] == 'i')
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1983 goto LscanReal; // 0Li
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1984 break; // 0L
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1985 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1986 if (p[1] == '.')
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1987 break; // 0..
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1988 // 0.
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1989 case 'i','f','F', // Imaginary and float literal suffixes.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1990 'e', 'E': // Float exponent.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1991 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1992 default:
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1993 if (*p == '_')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1994 goto LscanOctal; // 0_
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1995 else if (isdigit(*p))
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1996 {
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1997 if (p == '8' \|\| p == '9')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1998 goto Loctal_hasDecimalDigits; // 08 or 09
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1999 else
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2000 goto Loctal_enter_loop; // 0[0-7]
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2001 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2002 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2003
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2004 // Number 0
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2005 assert(p[-1] == '0');
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2006 assert(p != '_' && !isdigit(p));
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2007 assert(ulong_ == 0);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2008 isDecimal = true;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2009 goto Lfinalize;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2010
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2011 LscanInteger:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2012 assert(p != 0 && isdigit(p));
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2013 isDecimal = true;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2014 goto Lenter_loop_int;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2015 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2016 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2017 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2018 continue;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2019 if (!isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2020 break;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2021 Lenter_loop_int:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2022 if (ulong_ < ulong.max/10 \|\| (ulong_ == ulong.max/10 && *p <= '5'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2023 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2024 ulong_ *= 10;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2025 ulong_ += *p - '0';
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2026 continue;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2027 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2028 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2029 overflow = true;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2030 while (isdigit(*++p)) {}
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2031 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2032 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2033
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	2034 // The number could be a float, so check overflow below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2035 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2036 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2037 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2038 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2039 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2040 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2041 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2042 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2043 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2044 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2045 goto LscanReal;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2046 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2047 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2048
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2049 if (overflow)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2050 error(t.start, MID.OverflowDecimalNumber);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2051
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2052 assert((isdigit(p[-1]) \|\| p[-1] == '_') && !isdigit(p) && p != '_');
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2053 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2054
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2055 LscanHex:
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	2056 assert(digits == 0);
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	2057 assert(p == 'x' \|\| p == 'X');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2058 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2059 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2060 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2061 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2062 if (!ishexad(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2063 break;
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	2064 ++digits;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2065 ulong_ *= 16;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2066 if (*p <= '9')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2067 ulong_ += *p - '0';
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2068 else if (*p <= 'F')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2069 ulong_ += *p - 'A' + 10;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2070 else
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2071 ulong_ += *p - 'a' + 10;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2072 }
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2073
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	2074 assert(ishexad(p[-1]) \|\| p[-1] == '_' \|\| p[-1] == 'x' \|\| p[-1] == 'X');
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2075 assert(!ishexad(p) && p != '_');
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2076
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2077 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2078 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2079 case '.':
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2080 if (p[1] == '.')
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2081 break;
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2082 case 'p', 'P':
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2083 return scanHexReal(t);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2084 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2085 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2086
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2087 if (digits == 0 \|\| digits > 16)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2088 error(t.start, digits == 0 ? MID.NoDigitsInHexNumber : MID.OverflowHexNumber);
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2089
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2090 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2091
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2092 LscanBinary:
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2093 assert(digits == 0);
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	2094 assert(p == 'b' \|\| p == 'B');
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2095 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2096 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2097 if (*++p == '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2098 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2099 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2100 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2101 }
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2102 else if (*p == '1')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2103 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2104 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2105 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2106 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2107 }
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2108 else if (*p == '_')
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2109 continue;
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2110 else
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2111 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2112 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2113
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2114 if (digits == 0 \|\| digits > 64)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2115 error(t.start, digits == 0 ? MID.NoDigitsInBinNumber : MID.OverflowBinaryNumber);
59 3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	2116
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	2117 assert(p[-1] == '0' \|\| p[-1] == '1' \|\| p[-1] == '_' \|\| p[-1] == 'b' \|\| p[-1] == 'B', p[-1] ~ "");
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2118 assert( !(p == '0' \|\| p == '1' \|\| *p == '_') );
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2119 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2120
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2121 LscanOctal:
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2122 assert(*p == '_');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2123 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2124 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2125 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2126 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2127 if (!isoctal(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2128 break;
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2129 Loctal_enter_loop:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2130 if (ulong_ < ulong.max/2 \|\| (ulong_ == ulong.max/2 && *p <= '1'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2131 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2132 ulong_ *= 8;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2133 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2134 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2135 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2136 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2137 overflow = true;
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2138 while (isoctal(*++p)) {}
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2139 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2140 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2141
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2142 bool hasDecimalDigits;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2143 if (isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2144 {
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2145 Loctal_hasDecimalDigits:
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2146 hasDecimalDigits = true;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2147 while (isdigit(*++p)) {}
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2148 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2149
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2150 // The number could be a float, so check errors below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2151 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2152 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2153 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2154 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2155 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2156 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2157 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2158 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2159 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2160 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2161 goto LscanReal;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2162 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2163 }
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2164
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2165 if (hasDecimalDigits)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2166 error(t.start, MID.OctalNumberHasDecimals);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2167
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2168 if (overflow)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2169 error(t.start, MID.OverflowOctalNumber);
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2170 // goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2171
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2172 Lfinalize:
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2173 enum Suffix
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2174 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2175 None = 0,
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2176 Unsigned = 1,
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2177 Long = 2
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2178 }
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2179
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2180 // Scan optional suffix: L, Lu, LU, u, uL, U or UL.
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2181 Suffix suffix;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2182 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2183 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2184 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2185 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2186 case 'L':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2187 if (suffix & Suffix.Long)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2188 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2189 suffix \|= Suffix.Long;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2190 ++p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2191 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2192 case 'u', 'U':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2193 if (suffix & Suffix.Unsigned)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2194 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2195 suffix \|= Suffix.Unsigned;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2196 ++p;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2197 continue;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2198 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2199 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2200 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2201 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2202 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2203
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2204 // Determine type of Integer.
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2205 switch (suffix)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2206 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2207 case Suffix.None:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2208 if (ulong_ & 0x8000_0000_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2209 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2210 if (isDecimal)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2211 error(t.start, MID.OverflowDecimalSign);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2212 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2213 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2214 else if (ulong_ & 0xFFFF_FFFF_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2215 t.type = TOK.Int64;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2216 else if (ulong_ & 0x8000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2217 t.type = isDecimal ? TOK.Int64 : TOK.Uint32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2218 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2219 t.type = TOK.Int32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2220 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2221 case Suffix.Unsigned:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2222 if (ulong_ & 0xFFFF_FFFF_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2223 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2224 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2225 t.type = TOK.Uint32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2226 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2227 case Suffix.Long:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2228 if (ulong_ & 0x8000_0000_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2229 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2230 if (isDecimal)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2231 error(t.start, MID.OverflowDecimalSign);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2232 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2233 }
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2234 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2235 t.type = TOK.Int64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2236 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2237 case Suffix.Unsigned \| Suffix.Long:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2238 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2239 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2240 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2241 assert(0);
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2242 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2243 t.ulong_ = ulong_;
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	2244 t.end = p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2245 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2246 LscanReal:
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2247 scanReal(t);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2248 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2249 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2250
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2251 /*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2252 FloatLiteral:= Float[fFL]?i?
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2253 Float:= DecFloat \| HexFloat
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2254 DecFloat:= ([0-9][0-9_][.][0-9_]DecExponent?) \| [.][0-9][0-9_]DecExponent? \| [0-9][0-9_]DecExponent
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2255 DecExponent:= [eE][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2256 HexFloat:= 0[xX](HexDigits[.]HexDigits \| [.][0-9a-zA-Z]HexDigits? \| HexDigits)HexExponent
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2257 HexExponent:= [pP][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2258 */
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2259 void scanReal(ref Token t)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2260 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2261 if (*p == '.')
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2262 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2263 assert(p[1] != '.');
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2264 // This function was called by scan() or scanNumber().
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2265 while (isdigit(++p) \|\| p == '_') {}
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2266 }
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2267 else
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2268 // This function was called by scanNumber().
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2269 assert(delegate ()
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2270 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2271 switch (*p)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2272 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2273 case 'L':
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2274 if (p[1] != 'i')
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2275 return false;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2276 case 'i', 'f', 'F', 'e', 'E':
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2277 return true;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2278 default:
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2279 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2280 return false;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2281 }()
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2282 );
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2283
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2284 // Scan exponent.
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2285 if (p == 'e' \|\| p == 'E')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2286 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2287 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2288 if (p == '-' \|\| p == '+')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2289 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2290 if (isdigit(*p))
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2291 while (isdigit(++p) \|\| p == '_') {}
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2292 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2293 error(t.start, MID.FloatExpMustStartWithDigit);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2294 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2295
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2296 // Copy whole number and remove underscores from buffer.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2297 char[] buffer = t.start[0..p-t.start].dup;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2298 uint j;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2299 foreach (c; buffer)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2300 if (c != '_')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2301 buffer[j++] = c;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2302 buffer.length = j; // Adjust length.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2303 buffer ~= 0; // Terminate for C functions.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2304
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2305 finalizeFloat(t, buffer);
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2306 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2307
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2308 void scanHexReal(ref Token t)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2309 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2310 assert(p == '.' \|\| p == 'p' \|\| *p == 'P');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2311 MID mid;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2312 if (*p == '.')
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2313 while (ishexad(++p) \|\| p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2314 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2315 // Decimal exponent is required.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2316 if (p != 'p' && p != 'P')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2317 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2318 mid = MID.HexFloatExponentRequired;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2319 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2320 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2321 // Scan exponent
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2322 assert(p == 'p' \|\| p == 'P');
393 fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2323 ++p;
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2324 if (p == '+' \|\| p == '-')
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2325 ++p;
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2326 if (!isdigit(*p))
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2327 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2328 mid = MID.HexFloatExpMustStartWithDigit;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2329 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2330 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2331 while (isdigit(++p) \|\| p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2332 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2333 // Copy whole number and remove underscores from buffer.
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2334 char[] buffer = t.start[0..p-t.start].dup;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2335 uint j;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2336 foreach (c; buffer)
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2337 if (c != '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2338 buffer[j++] = c;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2339 buffer.length = j; // Adjust length.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2340 buffer ~= 0; // Terminate for C functions.
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2341 finalizeFloat(t, buffer);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2342 return;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2343 Lerr:
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2344 t.type = TOK.Float32;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2345 t.end = p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2346 error(t.start, mid);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2347 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2348
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2349 void finalizeFloat(ref Token t, string buffer)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2350 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2351 assert(buffer[$-1] == 0);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2352 // Float number is well-formed. Check suffixes and do conversion.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2353 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2354 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2355 case 'f', 'F':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2356 t.type = TOK.Float32;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2357 t.float_ = strtof(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2358 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2359 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2360 case 'L':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2361 t.type = TOK.Float80;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2362 t.real_ = strtold(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2363 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2364 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2365 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2366 t.type = TOK.Float64;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2367 t.double_ = strtod(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2368 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2369 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2370 if (*p == 'i')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2371 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2372 ++p;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2373 t.type += 3; // Switch to imaginary counterpart.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2374 }
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	2375 if (errno() == ERANGE)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2376 error(t.start, MID.OverflowFloatNumber);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2377 t.end = p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2378 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2379
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2380 /// Scan special token: #line Integer [Filespec] EndOfLine
360 b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence(). aziz parents: 350 diff changeset	2381 void scanSpecialTokenSequence(ref Token t)
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2382 {
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	2383 assert(*p == '#');
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2384 t.type = TOK.HashLine;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2385
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2386 MID mid;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2387 auto errorAtColumn = p;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2388
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2389 ++p;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2390 if (p[0] != 'l' \|\| p[1] != 'i' \|\| p[2] != 'n' \|\| p[3] != 'e')
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2391 {
370 ae4afb66768f - Renamed findModule() to findModulePath(). aziz parents: 367 diff changeset	2392 mid = MID.ExpectedIdentifierSTLine;
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2393 goto Lerr;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2394 }
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2395 p += 3;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2396
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2397 // TODO: #line58"path/file" is legal. Require spaces?
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2398 // State.Space could be used for that purpose.
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2399 enum State
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2400 { /+Space,+/ Integer, Filespec, End }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2401
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2402 State state = State.Integer;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2403
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2404 while (!isEndOfLine(++p))
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2405 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2406 if (isspace(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2407 continue;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2408 if (state == State.Integer)
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2409 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2410 if (!isdigit(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2411 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2412 errorAtColumn = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2413 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2414 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2415 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2416 t.tokLineNum = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2417 scan(*t.tokLineNum);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2418 if (t.tokLineNum.type != TOK.Int32 && t.tokLineNum.type != TOK.Uint32)
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2419 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2420 errorAtColumn = t.tokLineNum.start;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2421 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2422 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2423 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2424 --p; // Go one back because scan() advanced p past the integer.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2425 state = State.Filespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2426 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2427 else if (state == State.Filespec)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2428 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2429 if (*p != '"')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2430 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2431 errorAtColumn = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2432 mid = MID.ExpectedFilespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2433 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2434 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2435 t.tokLineFilespec = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2436 t.tokLineFilespec.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2437 t.tokLineFilespec.type = TOK.Filespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2438 while (*++p != '"')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2439 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2440 if (isEndOfLine(p))
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2441 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2442 errorAtColumn = t.tokLineFilespec.start;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2443 mid = MID.UnterminatedFilespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2444 t.tokLineFilespec.end = p;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2445 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2446 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2447 isascii(*p) \|\| decodeUTF8();
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2448 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2449 auto start = t.tokLineFilespec.start +1; // +1 skips '"'
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2450 t.tokLineFilespec.str = start[0 .. p - start];
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2451 t.tokLineFilespec.end = p + 1;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2452 state = State.End;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2453 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2454 else/+ if (state == State.End)+/
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2455 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2456 mid = MID.UnterminatedSpecialToken;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2457 goto Lerr;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2458 }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2459 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2460 assert(isEndOfLine(p));
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2461
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2462 if (state == State.Integer)
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2463 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2464 errorAtColumn = p;
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2465 mid = MID.ExpectedIntegerAfterSTLine;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2466 goto Lerr;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2467 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2468
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2469 // Evaluate #line only when not in token string.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2470 if (!inTokenString && t.tokLineNum)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2471 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2472 this.lineNum_hline = this.lineNum - t.tokLineNum.uint_ + 1;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2473 if (t.tokLineFilespec)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2474 this.errorPath = t.tokLineFilespec.str;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2475 }
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2476 t.end = p;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2477
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2478 return;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2479 Lerr:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2480 t.end = p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2481 error(errorAtColumn, mid);
371 01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	2482 }
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	2483
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2484 /++
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2485 Insert an empty dummy token before t.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2486 Useful in the parsing phase for representing a node in the AST
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2487 that doesn't consume an actual token from the source text.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2488 +/
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2489 Token* insertEmptyTokenBefore(Token* t)
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2490 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2491 assert(t !is null && t.prev !is null);
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2492 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type));
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2493 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type));
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2494
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2495 auto prev_t = t.prev;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2496 auto new_t = new Token;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2497 new_t.type = TOK.Empty;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2498 new_t.start = new_t.end = prev_t.end;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2499 // Link in new token.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2500 prev_t.next = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2501 new_t.prev = prev_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2502 new_t.next = t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2503 t.prev = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2504 return new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2505 }
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	2506
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2507 uint errorLineNumber(uint lineNum)
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	2508 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2509 return lineNum - this.lineNum_hline;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	2510 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	2511
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2512 void error(char* columnPos, MID mid, ...)
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2513 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2514 error_(this.lineNum, this.lineBegin, columnPos, mid, _arguments, _argptr);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2515 }
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2516
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2517 void error(uint lineNum, char* lineBegin, char* columnPos, MID mid, ...)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2518 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2519 error_(lineNum, lineBegin, columnPos, mid, _arguments, _argptr);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2520 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2521
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2522 void error_(uint lineNum, char* lineBegin, char* columnPos, MID mid,
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2523 TypeInfo[] _arguments, void* _argptr)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2524 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2525 lineNum = this.errorLineNumber(lineNum);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2526 auto location = new Location(errorPath, lineNum, lineBegin, columnPos);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2527 auto msg = Format(_arguments, _argptr, GetMsg(mid));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2528 errors ~= new Information(InfoType.Lexer, mid, location, msg);
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2529 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2530
306 9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2531 Token* getTokens()
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	2532 {
306 9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2533 while (nextToken() != TOK.EOF)
9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2534 {}
9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2535 return head;
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	2536 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2537
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2538 /// Scan the whole text until EOF is encountered.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2539 void scanAll()
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2540 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2541 while (nextToken() != TOK.EOF)
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2542 {}
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2543 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2544
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2545 /// HEAD -> Newline -> First Token
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2546 Token* firstToken()
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2547 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2548 return this.head.next.next;
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2549 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2550
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2551 static void loadKeywords(ref Identifier[string] table)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2552 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2553 foreach(k; keywords)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2554 table[k.str] = k;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2555 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2556
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2557 static bool isNonReservedIdentifier(char[] ident)
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2558 {
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2559 if (ident.length == 0)
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2560 return false;
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2561
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2562 static Identifier[string] reserved_ids_table;
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2563 if (reserved_ids_table is null)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2564 loadKeywords(reserved_ids_table);
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2565
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2566 size_t idx = 1; // Index to the 2nd character in ident.
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2567 dchar isFirstCharUniAlpha()
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2568 {
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2569 idx = 0;
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2570 // NB: decode() could throw an Exception which would be
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2571 // caught by the next try-catch-block.
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2572 return isUniAlpha(std.utf.decode(ident, idx));
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2573 }
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2574
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2575 try
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2576 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2577 if (isidbeg(ident[0]) \|\| !isascii(ident[0]) && isFirstCharUniAlpha())
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2578 {
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2579 foreach (dchar c; ident[idx..$])
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2580 if (!isident(c) && !isUniAlpha(c))
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2581 return false;
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2582 }
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2583 }
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2584 catch (Exception)
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2585 return false;
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2586
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2587 return !(ident in reserved_ids_table);
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2588 }
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2589
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2590 /++
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2591 Returns true if d can be encoded as a UTF-8 sequence.
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2592 +/
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2593 bool isEncodable(dchar d)
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2594 {
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2595 return d < 0xD800 \|\|
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2596 (d > 0xDFFF && d <= 0x10FFFF);
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2597 }
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2598
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2599 /++
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2600 There are a total of 66 noncharacters.
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2601 Returns true if this is one of them.
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2602 See_also: Chapter 16.7 Noncharacters in Unicode 5.0
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2603 +/
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2604 bool isNoncharacter(dchar d)
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2605 {
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2606 return 0xFDD0 <= d && d <= 0xFDEF \|\| // 32
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2607 d <= 0x10FFFF && (d & 0xFFFF) >= 0xFFFE; // 34
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2608 }
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2609
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2610 /++
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2611 Returns true if this character is not a noncharacter, not a surrogate
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2612 code point and not higher than 0x10FFFF.
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2613 +/
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2614 bool isValidDecodedChar(dchar d)
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2615 {
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2616 return d < 0xD800 \|\|
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2617 (d > 0xDFFF && d < 0xFDD0) \|\|
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2618 (d > 0xFDEF && d <= 0x10FFFF && (d & 0xFFFF) < 0xFFFE);
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2619 }
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2620
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2621 /// Is this a trail byte of a UTF-8 sequence?
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2622 bool isTrailByte(ubyte b)
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2623 {
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2624 return (b & 0xC0) == 0x80; // 10xx_xxxx
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2625 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2626
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2627 /// Is this a lead byte of a UTF-8 sequence?
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2628 bool isLeadByte(ubyte b)
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2629 {
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2630 return (b & 0xC0) == 0xC0; // 11xx_xxxx
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2631 }
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2632
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2633 dchar decodeUTF8()
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2634 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2635 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2636 char* p = this.p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2637 dchar d = *p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2638
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2639 ++p; // Move to second byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2640 // Error if second byte is not a trail byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2641 if (!isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2642 goto Lerr2;
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2643
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2644 // Check for overlong sequences.
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2645 switch (d)
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2646 {
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2647 case 0xE0, // 11100000 100xxxxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2648 0xF0, // 11110000 1000xxxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2649 0xF8, // 11111000 10000xxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2650 0xFC: // 11111100 100000xx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2651 if ((*p & d) == 0x80)
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2652 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2653 default:
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2654 if ((d & 0xFE) == 0xC0) // 1100000x
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2655 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2656 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2657
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2658 const char[] checkNextByte = "if (!isTrailByte(*++p))"
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2659 " goto Lerr2;";
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2660 const char[] appendSixBits = "d = (d << 6) \| *p & 0b0011_1111;";
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2661
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2662 // Decode
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2663 if ((d & 0b1110_0000) == 0b1100_0000)
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2664 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2665 // 110xxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2666 d &= 0b0001_1111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2667 mixin(appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2668 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2669 else if ((d & 0b1111_0000) == 0b1110_0000)
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2670 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2671 // 1110xxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2672 d &= 0b0000_1111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2673 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2674 checkNextByte ~ appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2675 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2676 else if ((d & 0b1111_1000) == 0b1111_0000)
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2677 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2678 // 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2679 d &= 0b0000_0111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2680 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2681 checkNextByte ~ appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2682 checkNextByte ~ appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2683 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2684 else
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2685 // 5 and 6 byte UTF-8 sequences are not allowed yet.
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2686 // 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2687 // 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2688 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2689
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2690 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2691
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2692 if (!isEncodable(d))
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2693 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2694 Lerr:
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2695 // Three cases:
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2696 // *) the UTF-8 sequence was successfully decoded but the resulting
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2697 // character is invalid.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2698 // p points to last trail byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2699 // *) the UTF-8 sequence is overlong.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2700 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2701 // *) the UTF-8 sequence has more than 4 bytes or starts with
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2702 // a trail byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2703 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2704 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2705 // Move to next ASCII character or lead byte of a UTF-8 sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2706 while (p < (end-1) && isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2707 ++p;
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2708 --p;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2709 assert(!isTrailByte(p[1]));
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2710 Lerr2:
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2711 d = REPLACEMENT_CHAR;
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2712 error(this.p, MID.InvalidUTF8Sequence);
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2713 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2714
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2715 this.p = p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2716 return d;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2717 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2718
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2719 private void encodeUTF8(ref char[] str, dchar d)
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2720 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2721 char[6] b;
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2722 assert(!isascii(d), "check for ASCII char before calling encodeUTF8().");
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2723 assert(isEncodable(d), "check that 'd' is encodable before calling encodeUTF8().");
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2724
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2725 if (d < 0x800)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2726 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2727 b[0] = 0xC0 \| (d >> 6);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2728 b[1] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2729 str ~= b[0..2];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2730 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2731 else if (d < 0x10000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2732 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2733 b[0] = 0xE0 \| (d >> 12);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2734 b[1] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2735 b[2] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2736 str ~= b[0..3];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2737 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2738 else if (d < 0x200000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2739 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2740 b[0] = 0xF0 \| (d >> 18);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2741 b[1] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2742 b[2] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2743 b[3] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2744 str ~= b[0..4];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2745 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2746 /+ // There are no 5 and 6 byte UTF-8 sequences yet.
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2747 else if (d < 0x4000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2748 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2749 b[0] = 0xF8 \| (d >> 24);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2750 b[1] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2751 b[2] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2752 b[3] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2753 b[4] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2754 str ~= b[0..5];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2755 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2756 else if (d < 0x80000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2757 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2758 b[0] = 0xFC \| (d >> 30);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2759 b[1] = 0x80 \| ((d >> 24) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2760 b[2] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2761 b[3] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2762 b[4] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2763 b[5] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2764 str ~= b[0..6];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2765 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2766 +/
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2767 else
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2768 assert(0);
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2769 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	2770 }
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2771
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2772 unittest
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2773 {
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	2774 Stdout("Testing Lexer.\n");
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2775 struct Pair
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2776 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2777 char[] tokenText;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2778 TOK type;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2779 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2780 static Pair[] pairs = [
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2781 {"//çay", TOK.Comment}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2782 {"&", TOK.AndBinary},
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2783 {"/çağ/", TOK.Comment}, {"&&", TOK.AndLogical},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2784 {"/+çak+/", TOK.Comment}, {"&=", TOK.AndAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2785 {">", TOK.Greater}, {"+", TOK.Plus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2786 {">=", TOK.GreaterEqual}, {"++", TOK.PlusPlus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2787 {">>", TOK.RShift}, {"+=", TOK.PlusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2788 {">>=", TOK.RShiftAssign}, {"-", TOK.Minus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2789 {">>>", TOK.URShift}, {"--", TOK.MinusMinus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2790 {">>>=", TOK.URShiftAssign}, {"-=", TOK.MinusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2791 {"<", TOK.Less}, {"=", TOK.Assign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2792 {"<=", TOK.LessEqual}, {"==", TOK.Equal},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2793 {"<>", TOK.LorG}, {"~", TOK.Tilde},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2794 {"<>=", TOK.LorEorG}, {"~=", TOK.CatAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2795 {"<<", TOK.LShift}, {"*", TOK.Mul},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2796 {"<<=", TOK.LShiftAssign}, {"*=", TOK.MulAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2797 {"!", TOK.Not}, {"/", TOK.Div},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2798 {"!=", TOK.NotEqual}, {"/=", TOK.DivAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2799 {"!<", TOK.UorGorE}, {"^", TOK.Xor},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2800 {"!>", TOK.UorLorE}, {"^=", TOK.XorAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2801 {"!<=", TOK.UorG}, {"%", TOK.Mod},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2802 {"!>=", TOK.UorL}, {"%=", TOK.ModAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2803 {"!<>", TOK.UorE}, {"(", TOK.LParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2804 {"!<>=", TOK.Unordered}, {")", TOK.RParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2805 {".", TOK.Dot}, {"[", TOK.LBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2806 {"..", TOK.Slice}, {"]", TOK.RBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2807 {"...", TOK.Ellipses}, {"{", TOK.LBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2808 {"\|", TOK.OrBinary}, {"}", TOK.RBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2809 {"\|\|", TOK.OrLogical}, {":", TOK.Colon},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2810 {"\|=", TOK.OrAssign}, {";", TOK.Semicolon},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2811 {"?", TOK.Question}, {",", TOK.Comma},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2812 {"$", TOK.Dollar}, {"cam", TOK.Identifier},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2813 {"çay", TOK.Identifier}, {".0", TOK.Float64},
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2814 {"0", TOK.Int32}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2815 {"\r", TOK.Newline}, {"\r\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2816 {"\u2028", TOK.Newline}, {"\u2029", TOK.Newline}
40 9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2817 ];
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2818
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2819 char[] src;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2820
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2821 // Join all token texts into a single string.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2822 foreach (i, pair; pairs)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2823 if (pair.type == TOK.Comment && pair.tokenText[1] == '/') // Line comment.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2824 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2825 assert(pairs[i+1].type == TOK.Newline); // Must be followed by a newline.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2826 src ~= pair.tokenText;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2827 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2828 else
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2829 src ~= pair.tokenText ~ " ";
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2830
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2831 auto lx = new Lexer(src, "");
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2832 auto token = lx.getTokens();
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2833
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2834 uint i;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2835 assert(token == lx.head);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2836 assert(token.next.type == TOK.Newline);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2837 token = token.next.next;
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2838 do
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2839 {
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2840 assert(i < pairs.length);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2841 assert(token.srcText == pairs[i].tokenText, Format("Scanned '{0}' but expected '{1}'", token.srcText, pairs[i].tokenText));
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2842 ++i;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2843 token = token.next;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2844 } while (token.type != TOK.EOF)
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	2845 }
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2846
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2847 unittest
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2848 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2849 Stdout("Testing method Lexer.peek()\n");
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2850 string sourceText = "unittest { }";
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2851 auto lx = new Lexer(sourceText, null);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2852
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2853 auto next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2854 lx.peek(next);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2855 assert(next.type == TOK.Newline);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2856 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2857 assert(next.type == TOK.Unittest);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2858 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2859 assert(next.type == TOK.LBrace);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2860 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2861 assert(next.type == TOK.RBrace);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2862 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2863 assert(next.type == TOK.EOF);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2864
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2865 lx = new Lexer("", null);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2866 next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2867 lx.peek(next);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2868 assert(next.type == TOK.Newline);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2869 lx.peek(next);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2870 assert(next.type == TOK.EOF);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2871 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2872
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2873 unittest
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2874 {
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2875 // Numbers unittest
68 7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2876 // 0L 0ULi 0_L 0_UL 0x0U 0x0p2 0_Fi 0_e2 0_F 0_i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2877 // 0u 0U 0uL 0UL 0L 0LU 0Lu
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2878 // 0Li 0f 0F 0fi 0Fi 0i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2879 // 0b_1_LU 0b1000u
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2880 // 0x232Lu
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2881 }
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2882
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2883 /// ASCII character properties table.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2884 static const int ptable[256] = [
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2885 0, 0, 0, 0, 0, 0, 0, 0, 0,32, 0,32,32, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2886 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2887 32, 0, 0x2200, 0, 0, 0, 0, 0x2700, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2888 7, 7, 7, 7, 7, 7, 7, 7, 6, 6, 0, 0, 0, 0, 0, 0x3f00,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2889 0,12,12,12,12,12,12, 8, 8, 8, 8, 8, 8, 8, 8, 8,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2890 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0, 0x5c00, 0, 0,16,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2891 0, 0x70c, 0x80c,12,12,12, 0xc0c, 8, 8, 8, 8, 8, 8, 8, 0xa08, 8,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2892 8, 8, 0xd08, 8, 0x908, 8, 0xb08, 8, 8, 8, 8, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2893 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2894 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2895 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2896 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2897 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2898 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2899 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2900 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2901 ];
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2902
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2903 enum CProperty
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2904 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2905 Octal = 1,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2906 Digit = 1<<1,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2907 Hex = 1<<2,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2908 Alpha = 1<<3,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2909 Underscore = 1<<4,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2910 Whitespace = 1<<5
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2911 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2912
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2913 const uint EVMask = 0xFF00; // Bit mask for escape value
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2914
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2915 private alias CProperty CP;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2916 int isoctal(char c) { return ptable[c] & CP.Octal; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2917 int isdigit(char c) { return ptable[c] & CP.Digit; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2918 int ishexad(char c) { return ptable[c] & CP.Hex; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2919 int isalpha(char c) { return ptable[c] & CP.Alpha; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2920 int isalnum(char c) { return ptable[c] & (CP.Alpha \| CP.Digit); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2921 int isidbeg(char c) { return ptable[c] & (CP.Alpha \| CP.Underscore); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2922 int isident(char c) { return ptable[c] & (CP.Alpha \| CP.Underscore \| CP.Digit); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2923 int isspace(char c) { return ptable[c] & CP.Whitespace; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2924 int char2ev(char c) { return ptable[c] >> 8; /(ptable[c] & EVMask) >> 8;/ }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2925 int isascii(uint c) { return c < 128; }
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2926
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2927 version(gen_ptable)
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2928 static this()
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2929 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2930 alias ptable p;
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2931 assert(p.length == 256);
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2932 // Initialize character properties table.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2933 for (int i; i < p.length; ++i)
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2934 {
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2935 p[i] = 0; // Reset
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2936 if ('0' <= i && i <= '7')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2937 p[i] \|= CP.Octal;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2938 if ('0' <= i && i <= '9')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2939 p[i] \|= CP.Digit;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2940 if (isdigit(i) \|\| 'a' <= i && i <= 'f' \|\| 'A' <= i && i <= 'F')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2941 p[i] \|= CP.Hex;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2942 if ('a' <= i && i <= 'z' \|\| 'A' <= i && i <= 'Z')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2943 p[i] \|= CP.Alpha;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2944 if (i == '_')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2945 p[i] \|= CP.Underscore;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2946 if (i == ' ' \|\| i == '\t' \|\| i == '\v' \|\| i == '\f')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2947 p[i] \|= CP.Whitespace;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2948 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2949 // Store escape sequence values in second byte.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2950 assert(CProperty.max <= ubyte.max, "character property flags and escape value byte overlap.");
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2951 p['\''] \|= 39 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2952 p['"'] \|= 34 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2953 p['?'] \|= 63 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2954 p['\\'] \|= 92 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2955 p['a'] \|= 7 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2956 p['b'] \|= 8 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2957 p['f'] \|= 12 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2958 p['n'] \|= 10 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2959 p['r'] \|= 13 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2960 p['t'] \|= 9 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2961 p['v'] \|= 11 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2962 // Print a formatted array literal.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2963 char[] array = "[\n";
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2964 foreach (i, c; ptable)
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2965 {
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2966 array ~= Format((c>255?" 0x{0:x},":"{0,2},"), c) ~ (((i+1) % 16) ? "":"\n");
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2967 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2968 array[$-2..$] = "\n]";
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2969 Stdout(array).newline;
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2970 }

Mercurial > projects > dil

annotate trunk/src/dil/Lexer.d @ 489:a7291d3ee9d7