projects/dil: trunk/src/dil/Lexer.d annotate

annotate trunk/src/dil/Lexer.d @ 494:9a7ca8c56e59

Refactored a few things in the Parser. Removed unnecessary 'bool hasBody' parameter from some declarations. Added shebang to list of tokens in Lexer unittest. Added some semantic methods.

author	Aziz K?ksal <aziz.koeksal@gmail.com>
date	Fri, 07 Dec 2007 18:22:35 +0100
parents	47be6bfe39cd
children	5a607597dc22

rev	line source
0 8ba2570de175 Initial import. aziz parents: diff changeset	1 /++
8ba2570de175 Initial import. aziz parents: diff changeset	2 Author: Aziz Köksal
249 32d354584b28 - Upgraded license notices to GPL3. aziz parents: 239 diff changeset	3 License: GPL3
0 8ba2570de175 Initial import. aziz parents: diff changeset	4 +/
326 4a7359b88c11 - Added package 'dil' to module declarations. aziz parents: 325 diff changeset	5 module dil.Lexer;
327 a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	6 import dil.Token;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	7 import dil.Information;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	8 import dil.Keywords;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	9 import dil.Identifier;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	10 import dil.Messages;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	11 import dil.HtmlEntities;
464 325714d8aa6c Added new module with compiler constants to resolve import cycle. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 434 diff changeset	12 import dil.CompilerInfo;
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	13 import tango.stdc.stdlib : strtof, strtod, strtold;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	14 import tango.stdc.errno : errno, ERANGE;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	15 import tango.stdc.time : time_t, time, ctime;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	16 import tango.stdc.string : strlen;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	17 import std.utf;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	18 import std.uni;
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	19 import common;
0 8ba2570de175 Initial import. aziz parents: diff changeset	20
490 47be6bfe39cd Refactored code and added new modules. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 489 diff changeset	21 public import dil.LexerFuncs;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	22
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	23 /// U+FFFD = �. Used to replace invalid Unicode characters.
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	24 const dchar REPLACEMENT_CHAR = '\uFFFD';
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	25
0 8ba2570de175 Initial import. aziz parents: diff changeset	26 class Lexer
8ba2570de175 Initial import. aziz parents: diff changeset	27 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	28 Token* head; /// The head of the doubly linked token list.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	29 Token* tail; /// The tail of the linked list. Set in scan().
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	30 Token* token; /// Points to the current token in the token list.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	31 string text; /// The source text.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	32 char[] filePath; /// Path to the source text.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	33 char* p; /// Points to the current character in the source text.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	34 char* end; /// Points one character past the end of the source text.
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	35
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	36 // Members used for error messages:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	37 Information[] errors;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	38 /// Always points to the beginning of the current line.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	39 char* lineBegin;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	40 // Token* newline; /// Current newline token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	41 uint lineNum = 1; /// Current, actual source text line number.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	42 uint lineNum_hline; /// Line number set by #line.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	43 uint inTokenString; /// > 0 if inside q{ }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	44 char[] errorPath; /// The path displayed in error messages.
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	45
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	46 Identifier[string] idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	47
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	48 /++
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	49 Construct a Lexer object.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	50 Params:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	51 text = the UTF-8 source code.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	52 filePath = the path to the source code; used for error messages.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	53 +/
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	54 this(string text, string filePath)
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	55 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	56 this.filePath = this.errorPath = filePath;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	57
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	58 this.text = text;
397 c99f8aeb7b4a Empty source files are handled correctly now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 396 diff changeset	59 if (text.length == 0 \|\| text[$-1] != 0)
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	60 {
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	61 this.text.length = this.text.length + 1;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	62 this.text[$-1] = 0;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	63 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	64
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	65 this.p = this.text.ptr;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	66 this.end = this.p + this.text.length;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	67 this.lineBegin = this.p;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	68 loadKeywords(this.idtable);
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	69
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	70 this.head = new Token;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	71 this.head.type = TOK.HEAD;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	72 this.head.start = this.head.end = this.p;
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	73 this.token = this.head;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	74 // Add a newline as the first token after the head.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	75 auto newline = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	76 newline.type = TOK.Newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	77 newline.start = newline.end = this.p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	78 newline.filePath = this.errorPath;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	79 newline.lineNum = 1;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	80 newline.lineNum_hline = 0;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	81 // Link in.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	82 this.token.next = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	83 newline.prev = this.token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	84 this.token = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	85 // this.newline = newline;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	86 scanShebang();
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	87 }
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	88
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	89 ~this()
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	90 {
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	91 auto token = head.next;
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	92 while (token !is null)
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	93 {
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	94 assert(token.type == TOK.EOF ? token == tail && token.next is null : 1);
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	95 delete token.prev;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	96 token = token.next;
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	97 }
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	98 delete tail;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	99 }
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	100
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	101 /++
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	102 The "shebang" may optionally appear once at the beginning of a file.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	103 Regexp: #![^\EndOfLine]*
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	104 +/
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	105 void scanShebang()
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	106 {
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	107 if (*p == '#' && p[1] == '!')
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	108 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	109 auto t = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	110 t.type = TOK.Shebang;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	111 t.start = p;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	112 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	113 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	114 isascii(*p) \|\| decodeUTF8();
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	115 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	116 this.token.next = t;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	117 t.prev = this.token;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	118 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	119 }
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	120
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	121 void finalizeSpecialToken(ref Token t)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	122 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	123 assert(t.srcText[0..2] == "__");
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	124 switch (t.type)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	125 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	126 case TOK.FILE:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	127 t.str = this.errorPath;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	128 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	129 case TOK.LINE:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	130 t.uint_ = this.errorLineNumber(this.lineNum);
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	131 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	132 case TOK.DATE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	133 TOK.TIME,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	134 TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	135 time_t time_val;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	136 time(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	137 char* str = ctime(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	138 char[] time_str = str[0 .. strlen(str)];
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	139 switch (t.type)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	140 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	141 case TOK.DATE:
346 ce9e2e77743f - Fix: include one space when slicing time_str for __DATE__. aziz parents: 344 diff changeset	142 time_str = time_str[4..11] ~ time_str[20..24] ~ \0; break;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	143 case TOK.TIME:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	144 time_str = time_str[11..19] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	145 case TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	146 time_str = time_str[0..24] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	147 default: assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	148 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	149 t.str = time_str;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	150 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	151 case TOK.VENDOR:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	152 t.str = VENDOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	153 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	154 case TOK.VERSION:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	155 t.uint_ = VERSION_MAJOR*1000 + VERSION_MINOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	156 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	157 default:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	158 assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	159 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	160 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	161
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	162 private void setLineBegin(char* p)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	163 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	164 // Check that we can look behind one character.
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	165 assert((p-1) >= text.ptr && p < end);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	166 // Check that previous character is a newline.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	167 assert(isNewlineEnd(p - 1));
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	168 this.lineBegin = p;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	169 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	170
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	171 private void scanNext(ref Token* t)
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	172 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	173 assert(t !is null);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	174 if (t.next)
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	175 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	176 t = t.next;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	177 // if (t.type == TOK.Newline)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	178 // this.newline = t;
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	179 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	180 else if (t != this.tail)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	181 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	182 Token* new_t = new Token;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	183 scan(*new_t);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	184 new_t.prev = t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	185 t.next = new_t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	186 t = new_t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	187 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	188 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	189
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	190 /// Advance t one token forward.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	191 void peek(ref Token* t)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	192 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	193 scanNext(t);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	194 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	195
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	196 /// Advance to the next token in the source text.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	197 TOK nextToken()
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	198 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	199 scanNext(this.token);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	200 return this.token.type;
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	201 }
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	202
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	203 /// Returns true if p points to the last character of a Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	204 bool isNewlineEnd(char* p)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	205 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	206 if (p == '\n' \|\| p == '\r')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	207 return true;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	208 if (p == LS[2] \|\| p == PS[2])
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	209 if ((p-2) >= text.ptr)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	210 if (p[-1] == LS[1] && p[-2] == LS[0])
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	211 return true;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	212 return false;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	213 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	214
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	215 /++
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	216 This is the old scan method.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	217 TODO: profile old and new to see which one is faster.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	218 +/
489 a7291d3ee9d7 Refactored classes that inherit from Node. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	219 public void scan(out Token t)
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	220 in
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	221 {
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	222 assert(text.ptr <= p && p < end);
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	223 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	224 out
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	225 {
397 c99f8aeb7b4a Empty source files are handled correctly now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 396 diff changeset	226 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type));
c99f8aeb7b4a Empty source files are handled correctly now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 396 diff changeset	227 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type));
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	228 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	229 body
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	230 {
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	231 // Scan whitespace.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	232 if (isspace(*p))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	233 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	234 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	235 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	236 {}
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	237 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	238
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	239 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	240 uint c = *p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	241 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	242 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	243 // Newline.
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	244 switch (*p)
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	245 {
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	246 case '\r':
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	247 if (p[1] == '\n')
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	248 ++p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	249 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	250 assert(isNewlineEnd(p));
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	251 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	252 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	253 setLineBegin(p);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	254 // this.newline = &t;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	255 t.type = TOK.Newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	256 t.filePath = this.errorPath;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	257 t.lineNum = lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	258 t.lineNum_hline = lineNum_hline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	259 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	260 return;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	261 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	262 if (isUnicodeNewline(p))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	263 {
412 fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	264 ++p; ++p;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	265 goto case '\n';
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	266 }
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	267 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	268 // Identifier or string literal.
13 e5211758b63c - Added isidbeg() function. aziz parents: 12 diff changeset	269 if (isidbeg(c))
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	270 {
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	271 if (c == 'r' && p[1] == '"' && ++p)
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	272 return scanRawStringLiteral(t);
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	273 if (c == 'x' && p[1] == '"')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	274 return scanHexStringLiteral(t);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	275 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	276 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	277 if (c == 'q' && p[1] == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	278 return scanDelimitedStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	279 if (c == 'q' && p[1] == '{')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	280 return scanTokenStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	281 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	282 // Scan identifier.
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	283 Lidentifier:
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	284 do
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	285 { c = *++p; }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	286 while (isident(c) \|\| !isascii(c) && isUniAlpha(decodeUTF8()))
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	287
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	288 t.end = p;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	289
65 6c21ae79fbb3 - Renamed function Token.span to Token.srcText. aziz parents: 64 diff changeset	290 string str = t.srcText;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	291 Identifier* id = str in idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	292
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	293 if (!id)
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	294 {
327 a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	295 idtable[str] = Identifier(TOK.Identifier, str);
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	296 id = str in idtable;
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	297 }
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	298 assert(id);
3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	299 t.type = id.type;
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	300 if (t.type == TOK.Identifier)
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	301 return;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	302 if (t.type == TOK.EOF)
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	303 {
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	304 t.type = TOK.EOF;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	305 t.end = p;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	306 tail = &t;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	307 assert(t.srcText == "__EOF__");
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	308 }
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	309 else if (t.isSpecialToken)
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	310 finalizeSpecialToken(t);
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	311 return;
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	312 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	313
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	314 if (isdigit(c))
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	315 return scanNumber(t);
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	316
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	317 if (c == '/')
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	318 {
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	319 c = *++p;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	320 switch(c)
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	321 {
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	322 case '=':
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	323 ++p;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	324 t.type = TOK.DivAssign;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	325 t.end = p;
cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	326 return;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	327 case '+':
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	328 return scanNestedComment(t);
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	329 case '*':
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	330 return scanBlockComment(t);
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	331 case '/':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	332 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	333 isascii(*p) \|\| decodeUTF8();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	334 t.type = TOK.Comment;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	335 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	336 return;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	337 default:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	338 t.type = TOK.Div;
10 3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	339 t.end = p;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	340 return;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	341 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	342 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	343
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	344 switch (c)
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	345 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	346 case '\'':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	347 return scanCharacterLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	348 case '`':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	349 return scanRawStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	350 case '"':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	351 return scanNormalStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	352 case '\\':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	353 char[] buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	354 do
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	355 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	356 c = scanEscapeSequence();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	357 if (isascii(c))
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	358 buffer ~= c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	359 else
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	360 encodeUTF8(buffer, c);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	361 } while (*p == '\\')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	362 buffer ~= 0;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	363 t.type = TOK.String;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	364 t.str = buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	365 t.end = p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	366 return;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	367 case '>': /* > >= >> >>= >>> >>>= */
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	368 c = *++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	369 switch (c)
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	370 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	371 case '=':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	372 t.type = TOK.GreaterEqual;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	373 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	374 case '>':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	375 if (p[1] == '>')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	376 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	377 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	378 if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	379 { ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	380 t.type = TOK.URShiftAssign;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	381 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	382 else
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	383 t.type = TOK.URShift;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	384 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	385 else if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	386 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	387 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	388 t.type = TOK.RShiftAssign;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	389 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	390 else
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	391 t.type = TOK.RShift;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	392 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	393 default:
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	394 t.type = TOK.Greater;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	395 goto Lcommon2;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	396 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	397 assert(0);
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	398 case '<': /* < <= <> <>= << <<= */
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	399 c = *++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	400 switch (c)
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	401 {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	402 case '=':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	403 t.type = TOK.LessEqual;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	404 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	405 case '<':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	406 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	407 ++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	408 t.type = TOK.LShiftAssign;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	409 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	410 else
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	411 t.type = TOK.LShift;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	412 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	413 case '>':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	414 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	415 ++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	416 t.type = TOK.LorEorG;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	417 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	418 else
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	419 t.type = TOK.LorG;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	420 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	421 default:
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	422 t.type = TOK.Less;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	423 goto Lcommon2;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	424 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	425 assert(0);
37 7f3bcb97d017 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 36 diff changeset	426 case '!': /* ! !< !> !<= !>= !<> !<>= */
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	427 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	428 switch (c)
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	429 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	430 case '<':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	431 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	432 if (c == '>')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	433 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	434 if (p[1] == '=') {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	435 ++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	436 t.type = TOK.Unordered;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	437 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	438 else
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	439 t.type = TOK.UorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	440 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	441 else if (c == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	442 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	443 t.type = TOK.UorG;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	444 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	445 else {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	446 t.type = TOK.UorGorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	447 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	448 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	449 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	450 case '>':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	451 if (p[1] == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	452 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	453 ++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	454 t.type = TOK.UorL;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	455 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	456 else
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	457 t.type = TOK.UorLorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	458 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	459 case '=':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	460 t.type = TOK.NotEqual;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	461 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	462 default:
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	463 t.type = TOK.Not;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	464 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	465 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	466 assert(0);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	467 case '.': /* . .[0-9] .. ... */
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	468 if (p[1] == '.')
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	469 {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	470 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	471 if (p[1] == '.') {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	472 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	473 t.type = TOK.Ellipses;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	474 }
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	475 else
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	476 t.type = TOK.Slice;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	477 }
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	478 else if (isdigit(p[1]))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	479 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	480 return scanReal(t);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	481 }
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	482 else
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	483 t.type = TOK.Dot;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	484 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	485 case '\|': /* \| \|\| \|= */
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	486 c = *++p;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	487 if (c == '=')
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	488 t.type = TOK.OrAssign;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	489 else if (c == '\|')
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	490 t.type = TOK.OrLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	491 else {
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	492 t.type = TOK.OrBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	493 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	494 }
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	495 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	496 case '&': /* & && &= */
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	497 c = *++p;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	498 if (c == '=')
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	499 t.type = TOK.AndAssign;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	500 else if (c == '&')
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	501 t.type = TOK.AndLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	502 else {
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	503 t.type = TOK.AndBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	504 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	505 }
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	506 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	507 case '+': /* + ++ += */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	508 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	509 if (c == '=')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	510 t.type = TOK.PlusAssign;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	511 else if (c == '+')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	512 t.type = TOK.PlusPlus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	513 else {
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	514 t.type = TOK.Plus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	515 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	516 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	517 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	518 case '-': /* - -- -= */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	519 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	520 if (c == '=')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	521 t.type = TOK.MinusAssign;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	522 else if (c == '-')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	523 t.type = TOK.MinusMinus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	524 else {
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	525 t.type = TOK.Minus;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	526 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	527 }
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	528 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	529 case '=': /* = == */
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	530 if (p[1] == '=') {
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	531 ++p;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	532 t.type = TOK.Equal;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	533 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	534 else
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	535 t.type = TOK.Assign;
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	536 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	537 case '~': /* ~ ~= */
27 43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	538 if (p[1] == '=') {
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	539 ++p;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	540 t.type = TOK.CatAssign;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	541 }
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	542 else
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	543 t.type = TOK.Tilde;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	544 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	545 case '': / * = /
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	546 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	547 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	548 t.type = TOK.MulAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	549 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	550 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	551 t.type = TOK.Mul;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	552 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	553 case '^': /* ^ ^= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	554 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	555 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	556 t.type = TOK.XorAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	557 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	558 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	559 t.type = TOK.Xor;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	560 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	561 case '%': /* % %= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	562 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	563 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	564 t.type = TOK.ModAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	565 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	566 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	567 t.type = TOK.Mod;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	568 goto Lcommon;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	569 // Single character tokens:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	570 case '(':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	571 t.type = TOK.LParen;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	572 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	573 case ')':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	574 t.type = TOK.RParen;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	575 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	576 case '[':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	577 t.type = TOK.LBracket;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	578 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	579 case ']':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	580 t.type = TOK.RBracket;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	581 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	582 case '{':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	583 t.type = TOK.LBrace;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	584 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	585 case '}':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	586 t.type = TOK.RBrace;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	587 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	588 case ':':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	589 t.type = TOK.Colon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	590 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	591 case ';':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	592 t.type = TOK.Semicolon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	593 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	594 case '?':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	595 t.type = TOK.Question;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	596 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	597 case ',':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	598 t.type = TOK.Comma;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	599 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	600 case '$':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	601 t.type = TOK.Dollar;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	602 Lcommon:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	603 ++p;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	604 Lcommon2:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	605 t.end = p;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	606 return;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	607 case '#':
360 b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence(). aziz parents: 350 diff changeset	608 return scanSpecialTokenSequence(t);
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	609 default:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	610 }
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	611
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	612 // Check for EOF
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	613 if (c == 0 \|\| c == _Z_)
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	614 {
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	615 assert(p == 0 \|\| p == _Z_);
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	616 t.type = TOK.EOF;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	617 t.end = p;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	618 tail = &t;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	619 assert(t.start == t.end);
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	620 return;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	621 }
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	622
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	623 if (!isascii(c))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	624 {
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	625 c = decodeUTF8();
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	626 if (isUniAlpha(c))
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	627 goto Lidentifier;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	628 }
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	629
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	630 error(t.start, MID.IllegalCharacter, cast(dchar)c);
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	631
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	632 ++p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	633 t.type = TOK.Illegal;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	634 t.dchar_ = c;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	635 t.end = p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	636 return;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	637 }
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	638 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	639
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	640 template toUint(char[] T)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	641 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	642 static assert(0 < T.length && T.length <= 4);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	643 static if (T.length == 1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	644 const uint toUint = T[0];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	645 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	646 const uint toUint = (T[0] << ((T.length-1)*8)) \| toUint!(T[1..$]);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	647 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	648 static assert(toUint!("\xAA\xBB\xCC\xDD") == 0xAABBCCDD);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	649
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	650 // Can't use this yet due to a bug in DMD (bug id=1534).
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	651 template case_(char[] str, TOK tok, char[] label)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	652 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	653 const char[] case_ =
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	654 `case `~toUint!(str).stringof~`:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	655
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	656 goto `~label~`;`;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	657 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	658
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	659 template case_L4(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	660 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	661 const char[] case_L4 = case_!(str, tok, "Lcommon_4");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	662 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	663
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	664 template case_L3(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	665 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	666 const char[] case_L3 = case_!(str, tok, "Lcommon_3");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	667 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	668
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	669 template case_L2(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	670 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	671 const char[] case_L2 = case_!(str, tok, "Lcommon_2");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	672 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	673
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	674 template case_L1(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	675 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	676 const char[] case_L3 = case_!(str, tok, "Lcommon");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	677 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	678
489 a7291d3ee9d7 Refactored classes that inherit from Node. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 487 diff changeset	679 public void scan_(out Token t)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	680 in
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	681 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	682 assert(text.ptr <= p && p < end);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	683 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	684 out
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	685 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	686 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type));
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	687 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type));
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	688 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	689 body
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	690 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	691 // Scan whitespace.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	692 if (isspace(*p))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	693 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	694 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	695 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	696 {}
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	697 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	698
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	699 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	700 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	701 // Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	702 switch (*p)
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	703 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	704 case '\r':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	705 if (p[1] == '\n')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	706 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	707 case '\n':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	708 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	709 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	710 ++lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	711 setLineBegin(p);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	712 // this.newline = &t;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	713 t.type = TOK.Newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	714 t.filePath = this.errorPath;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	715 t.lineNum = lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	716 t.lineNum_hline = lineNum_hline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	717 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	718 return;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	719 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	720 if (isUnicodeNewline(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	721 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	722 ++p; ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	723 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	724 }
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	725 }
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	726
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	727 uint c = *p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	728 assert(end - p != 0);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	729 switch (end - p)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	730 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	731 case 1:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	732 goto L1character;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	733 case 2:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	734 c <<= 8; c \|= p[1];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	735 goto L2characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	736 case 3:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	737 c <<= 8; c \|= p[1]; c <<= 8; c \|= p[2];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	738 goto L3characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	739 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	740 version(BigEndian)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	741 c = cast(uint)p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	742 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	743 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	744 c <<= 8; c \|= p[1]; c <<= 8; c \|= p[2]; c <<= 8; c \|= p[3];
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	745 /+
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	746 c = cast(uint)p;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	747 asm
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	748 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	749 mov EDX, c;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	750 bswap EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	751 mov c, EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	752 }
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	753 +/
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	754 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	755 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	756
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	757 // 4 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	758 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	759 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	760 case toUint!(">>>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	761 t.type = TOK.RShiftAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	762 goto Lcommon_4;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	763 case toUint!("!<>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	764 t.type = TOK.Unordered;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	765 Lcommon_4:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	766 p += 4;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	767 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	768 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	769 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	770 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	771
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	772 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	773 L3characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	774 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	775 // 3 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	776 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	777 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	778 case toUint!(">>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	779 t.type = TOK.RShiftAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	780 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	781 case toUint!(">>>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	782 t.type = TOK.URShift;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	783 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	784 case toUint!("<>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	785 t.type = TOK.LorEorG;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	786 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	787 case toUint!("<<="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	788 t.type = TOK.LShiftAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	789 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	790 case toUint!("!<="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	791 t.type = TOK.UorG;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	792 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	793 case toUint!("!>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	794 t.type = TOK.UorL;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	795 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	796 case toUint!("!<>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	797 t.type = TOK.UorE;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	798 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	799 case toUint!("..."):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	800 t.type = TOK.Ellipses;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	801 Lcommon_3:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	802 p += 3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	803 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	804 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	805 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	806 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	807
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	808 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	809 L2characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	810 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	811 // 2 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	812 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	813 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	814 case toUint!("/+"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	815 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	816 return scanNestedComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	817 case toUint!("/*"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	818 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	819 return scanBlockComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	820 case toUint!("//"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	821 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	822 assert(*p == '/');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	823 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	824 isascii(*p) \|\| decodeUTF8();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	825 t.type = TOK.Comment;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	826 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	827 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	828 case toUint!(">="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	829 t.type = TOK.GreaterEqual;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	830 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	831 case toUint!(">>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	832 t.type = TOK.RShift;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	833 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	834 case toUint!("<<"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	835 t.type = TOK.LShift;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	836 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	837 case toUint!("<="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	838 t.type = TOK.LessEqual;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	839 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	840 case toUint!("<>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	841 t.type = TOK.LorG;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	842 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	843 case toUint!("!<"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	844 t.type = TOK.UorGorE;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	845 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	846 case toUint!("!>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	847 t.type = TOK.UorLorE;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	848 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	849 case toUint!("!="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	850 t.type = TOK.NotEqual;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	851 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	852 case toUint!(".."):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	853 t.type = TOK.Slice;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	854 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	855 case toUint!("&&"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	856 t.type = TOK.AndLogical;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	857 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	858 case toUint!("&="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	859 t.type = TOK.AndAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	860 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	861 case toUint!("\|\|"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	862 t.type = TOK.OrLogical;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	863 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	864 case toUint!("\|="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	865 t.type = TOK.OrAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	866 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	867 case toUint!("++"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	868 t.type = TOK.PlusPlus;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	869 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	870 case toUint!("+="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	871 t.type = TOK.PlusAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	872 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	873 case toUint!("--"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	874 t.type = TOK.MinusMinus;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	875 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	876 case toUint!("-="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	877 t.type = TOK.MinusAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	878 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	879 case toUint!("=="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	880 t.type = TOK.Equal;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	881 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	882 case toUint!("~="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	883 t.type = TOK.CatAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	884 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	885 case toUint!("*="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	886 t.type = TOK.MulAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	887 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	888 case toUint!("/="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	889 t.type = TOK.DivAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	890 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	891 case toUint!("^="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	892 t.type = TOK.XorAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	893 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	894 case toUint!("%="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	895 t.type = TOK.ModAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	896 Lcommon_2:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	897 p += 2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	898 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	899 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	900 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	901 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	902
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	903 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	904 L1character:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	905 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	906 assert(p == c, Format("p={0},c={1}", p, cast(dchar)c));
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	907 // 1 character tokens.
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	908 // TODO: consider storing the token type in ptable.
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	909 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	910 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	911 case '\'':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	912 return scanCharacterLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	913 case '`':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	914 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	915 case '"':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	916 return scanNormalStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	917 case '\\':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	918 char[] buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	919 do
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	920 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	921 c = scanEscapeSequence();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	922 if (isascii(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	923 buffer ~= c;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	924 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	925 encodeUTF8(buffer, c);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	926 } while (*p == '\\')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	927 buffer ~= 0;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	928 t.type = TOK.String;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	929 t.str = buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	930 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	931 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	932 case '<':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	933 t.type = TOK.Greater;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	934 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	935 case '>':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	936 t.type = TOK.Less;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	937 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	938 case '^':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	939 t.type = TOK.Xor;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	940 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	941 case '!':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	942 t.type = TOK.Not;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	943 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	944 case '.':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	945 if (isdigit(p[1]))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	946 return scanReal(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	947 t.type = TOK.Dot;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	948 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	949 case '&':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	950 t.type = TOK.AndBinary;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	951 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	952 case '\|':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	953 t.type = TOK.OrBinary;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	954 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	955 case '+':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	956 t.type = TOK.Plus;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	957 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	958 case '-':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	959 t.type = TOK.Minus;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	960 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	961 case '=':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	962 t.type = TOK.Assign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	963 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	964 case '~':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	965 t.type = TOK.Tilde;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	966 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	967 case '*':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	968 t.type = TOK.Mul;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	969 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	970 case '/':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	971 t.type = TOK.Div;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	972 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	973 case '%':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	974 t.type = TOK.Mod;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	975 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	976 case '(':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	977 t.type = TOK.LParen;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	978 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	979 case ')':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	980 t.type = TOK.RParen;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	981 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	982 case '[':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	983 t.type = TOK.LBracket;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	984 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	985 case ']':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	986 t.type = TOK.RBracket;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	987 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	988 case '{':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	989 t.type = TOK.LBrace;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	990 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	991 case '}':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	992 t.type = TOK.RBrace;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	993 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	994 case ':':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	995 t.type = TOK.Colon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	996 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	997 case ';':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	998 t.type = TOK.Semicolon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	999 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1000 case '?':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1001 t.type = TOK.Question;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1002 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1003 case ',':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1004 t.type = TOK.Comma;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1005 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1006 case '$':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1007 t.type = TOK.Dollar;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1008 Lcommon:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1009 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1010 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1011 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1012 case '#':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1013 return scanSpecialTokenSequence(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1014 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1015 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1016
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1017 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1018 assert(*p == c);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1019
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1020 // TODO: consider moving isidbeg() and isdigit() up.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1021 if (isidbeg(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1022 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1023 if (c == 'r' && p[1] == '"' && ++p)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1024 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1025 if (c == 'x' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1026 return scanHexStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1027 version(D2)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1028 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1029 if (c == 'q' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1030 return scanDelimitedStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1031 if (c == 'q' && p[1] == '{')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1032 return scanTokenStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1033 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1034 // Scan identifier.
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1035 Lidentifier:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1036 do
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1037 { c = *++p; }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1038 while (isident(c) \|\| !isascii(c) && isUniAlpha(decodeUTF8()))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1039
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1040 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1041
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1042 string str = t.srcText;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1043 Identifier* id = str in idtable;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1044
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1045 if (!id)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1046 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1047 idtable[str] = Identifier(TOK.Identifier, str);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1048 id = str in idtable;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1049 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1050 assert(id);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1051 t.type = id.type;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1052 if (t.type == TOK.Identifier)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1053 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1054 if (t.type == TOK.EOF)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1055 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1056 t.type = TOK.EOF;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1057 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1058 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1059 assert(t.srcText == "__EOF__");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1060 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1061 else if (t.isSpecialToken)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1062 finalizeSpecialToken(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1063 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1064 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1065
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1066 if (isdigit(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1067 return scanNumber(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1068
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1069 // Check for EOF
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1070 if (c == 0 \|\| c == _Z_)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1071 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1072 assert(p == 0 \|\| p == _Z_, *p~"");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1073 t.type = TOK.EOF;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1074 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1075 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1076 assert(t.start == t.end);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1077 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1078 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1079
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1080 if (!isascii(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1081 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1082 c = decodeUTF8();
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1083 if (isUniAlpha(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1084 goto Lidentifier;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1085 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1086
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1087 error(t.start, MID.IllegalCharacter, cast(dchar)c);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1088
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1089 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1090 t.type = TOK.Illegal;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1091 t.dchar_ = c;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1092 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1093 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1094 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1095
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1096 void scanBlockComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1097 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1098 assert(p[-1] == '/' && p == '');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1099 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1100 auto tokenLineBegin = lineBegin;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1101 uint c;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1102 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1103 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1104 c = *++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1105 LswitchBC: // only jumped to from default case of next switch(c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1106 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1107 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1108 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1109 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1110 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1111 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1112 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1113 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1114 setLineBegin(p+1);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1115 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1116 case 0, _Z_:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1117 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedBlockComment);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1118 goto LreturnBC;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1119 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1120 if (!isascii(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1121 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1122 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1123 if (isUnicodeNewlineChar(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1124 goto case '\n';
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1125 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1126 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1127 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1128
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1129 c <<= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1130 c \|= *++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1131 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1132 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1133 case toUint!("*/"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1134 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1135 LreturnBC:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1136 t.type = TOK.Comment;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1137 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1138 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1139 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1140 c &= char.max;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1141 goto LswitchBC;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1142 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1143 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1144 assert(0);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1145 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1146
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1147 void scanNestedComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1148 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1149 assert(p[-1] == '/' && *p == '+');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1150 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1151 auto tokenLineBegin = lineBegin;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1152 uint level = 1;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1153 uint c;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1154 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1155 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1156 c = *++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1157 LswitchNC: // only jumped to from default case of next switch(c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1158 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1159 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1160 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1161 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1162 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1163 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1164 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1165 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1166 setLineBegin(p+1);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1167 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1168 case 0, _Z_:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1169 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedNestedComment);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1170 goto LreturnNC;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1171 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1172 if (!isascii(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1173 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1174 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1175 if (isUnicodeNewlineChar(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1176 goto case '\n';
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1177 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1178 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1179 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1180
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1181 c <<= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1182 c \|= *++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1183 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1184 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1185 case toUint!("/+"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1186 ++level;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1187 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1188 case toUint!("+/"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1189 if (--level == 0)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1190 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1191 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1192 LreturnNC:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1193 t.type = TOK.Comment;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1194 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1195 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1196 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1197 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1198 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1199 c &= char.max;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1200 goto LswitchNC;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1201 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1202 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1203 assert(0);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1204 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1205
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1206 char scanPostfix()
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1207 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1208 assert(p[-1] == '"' \|\| p[-1] == '`' \|\|
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1209 { version(D2) return p[-1] == '}';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1210 else return 0; }()
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1211 );
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1212 switch (*p)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1213 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1214 case 'c':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1215 case 'w':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1216 case 'd':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1217 return *p++;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1218 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1219 return 0;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1220 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1221 assert(0);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1222 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1223
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1224 void scanNormalStringLiteral(ref Token t)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1225 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1226 assert(*p == '"');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1227 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1228 auto tokenLineBegin = lineBegin;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1229 t.type = TOK.String;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1230 char[] buffer;
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1231 uint c;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1232 while (1)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1233 {
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1234 c = *++p;
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1235 switch (c)
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1236 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1237 case '"':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1238 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1239 t.pf = scanPostfix();
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1240 Lreturn:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1241 t.str = buffer ~ '\0';
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1242 t.end = p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1243 return;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1244 case '\\':
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1245 c = scanEscapeSequence();
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1246 --p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1247 if (isascii(c))
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1248 break;
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1249 encodeUTF8(buffer, c);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1250 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1251 case '\r':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1252 if (p[1] == '\n')
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1253 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1254 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1255 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1256 c = '\n'; // Convert Newline to \n.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1257 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1258 setLineBegin(p+1);
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1259 break;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1260 case 0, _Z_:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1261 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedString);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1262 goto Lreturn;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1263 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1264 if (!isascii(c))
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1265 {
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1266 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1267 if (isUnicodeNewlineChar(c))
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1268 goto case '\n';
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1269 encodeUTF8(buffer, c);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1270 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1271 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1272 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1273 assert(isascii(c));
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1274 buffer ~= c;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1275 }
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1276 assert(0);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1277 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1278
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1279 void scanCharacterLiteral(ref Token t)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1280 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1281 assert(*p == '\'');
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1282 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1283 t.type = TOK.CharLiteral;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1284 switch (*p)
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1285 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1286 case '\\':
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1287 switch (p[1])
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1288 {
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1289 case 'u':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1290 t.type = TOK.WCharLiteral; break;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1291 case 'U':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1292 t.type = TOK.DCharLiteral; break;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1293 default:
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1294 }
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1295 t.dchar_ = scanEscapeSequence();
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1296 break;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1297 case '\'':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1298 error(t.start, MID.EmptyCharacterLiteral);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1299 break;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1300 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1301 if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1302 break;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1303 uint c = *p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1304 if (!isascii(c))
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1305 {
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1306 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1307 t.type = c <= 0xFFFF ? TOK.WCharLiteral : TOK.DCharLiteral;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1308 }
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1309 t.dchar_ = c;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1310 ++p;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1311 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1312
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1313 if (*p == '\'')
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1314 ++p;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1315 else
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1316 error(t.start, MID.UnterminatedCharacterLiteral);
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1317 t.end = p;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1318 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1319
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1320 void scanRawStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1321 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1322 assert(p == '`' \|\| p == '"' && p[-1] == 'r');
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1323 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1324 auto tokenLineBegin = lineBegin;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1325 t.type = TOK.String;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1326 uint delim = *p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1327 char[] buffer;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1328 uint c;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1329 while (1)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1330 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1331 c = *++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1332 switch (c)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1333 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1334 case '\r':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1335 if (p[1] == '\n')
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1336 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1337 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1338 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1339 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1340 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1341 setLineBegin(p+1);
52 f65a83c27638 - Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well. aziz parents: 51 diff changeset	1342 break;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1343 case '`':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1344 case '"':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1345 if (c == delim)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1346 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1347 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1348 t.pf = scanPostfix();
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1349 Lreturn:
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1350 t.str = buffer ~ '\0';
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1351 t.end = p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1352 return;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1353 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1354 break;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1355 case 0, _Z_:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1356 error(tokenLineNum, tokenLineBegin, t.start,
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1357 delim == 'r' ? MID.UnterminatedRawString : MID.UnterminatedBackQuoteString);
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1358 goto Lreturn;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1359 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1360 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1361 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1362 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1363 if (isUnicodeNewlineChar(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1364 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1365 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1366 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1367 }
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1368 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1369 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1370 buffer ~= c;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1371 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1372 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1373 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1374
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1375 void scanHexStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1376 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1377 assert(p[0] == 'x' && p[1] == '"');
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1378 t.type = TOK.String;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1379
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1380 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1381 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1382
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1383 uint c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1384 ubyte[] buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1385 ubyte h; // hex number
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1386 uint n; // number of hex digits
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1387
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1388 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1389 assert(*p == '"');
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1390 while (1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1391 {
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1392 c = *++p;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1393 switch (c)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1394 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1395 case '"':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1396 if (n & 1)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1397 error(tokenLineNum, tokenLineBegin, t.start, MID.OddNumberOfDigitsInHexString);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1398 ++p;
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1399 t.pf = scanPostfix();
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1400 Lreturn:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1401 t.str = cast(string) (buffer ~= 0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1402 t.end = p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1403 return;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1404 case '\r':
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1405 if (p[1] == '\n')
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1406 ++p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1407 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1408 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1409 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1410 setLineBegin(p+1);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1411 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1412 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1413 if (ishexad(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1414 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1415 if (c <= '9')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1416 c -= '0';
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1417 else if (c <= 'F')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1418 c -= 'A' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1419 else
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1420 c -= 'a' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1421
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1422 if (n & 1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1423 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1424 h <<= 4;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1425 h \|= c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1426 buffer ~= h;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1427 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1428 else
112 004d98df65af - Implemented parseInterfaceDeclaration(). aziz parents: 105 diff changeset	1429 h = cast(ubyte)c;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1430 ++n;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1431 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1432 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1433 else if (isspace(c))
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1434 continue; // Skip spaces.
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1435 else if (c == 0 \|\| c == _Z_)
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1436 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1437 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedHexString);
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1438 t.pf = 0;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1439 goto Lreturn;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1440 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1441 else
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1442 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1443 auto errorAt = p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1444 if (!isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1445 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1446 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1447 if (isUnicodeNewlineChar(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1448 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1449 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1450 error(errorAt, MID.NonHexCharInHexString, cast(dchar)c);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1451 }
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1452 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1453 }
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1454 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1455 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1456
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1457 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1458 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1459 void scanDelimitedStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1460 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1461 assert(p[0] == 'q' && p[1] == '"');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1462 t.type = TOK.String;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1463
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1464 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1465 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1466
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1467 char[] buffer;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1468 dchar opening_delim = 0, // 0 if no nested delimiter or '[', '(', '<', '{'
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1469 closing_delim; // Will be ']', ')', '>', '},
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1470 // the first character of an identifier or
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1471 // any other Unicode/ASCII character.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1472 char[] str_delim; // Identifier delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1473 uint level = 1; // Counter for nestable delimiters.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1474
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1475 ++p; ++p; // Skip q"
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1476 uint c = *p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1477 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1478 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1479 case '(':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1480 opening_delim = c;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1481 closing_delim = ')'; // c + 1
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1482 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1483 case '[', '<', '{':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1484 opening_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1485 closing_delim = c + 2; // Get to closing counterpart. Feature of ASCII table.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1486 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1487 default:
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1488 dchar scanNewline()
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1489 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1490 switch (*p)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1491 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1492 case '\r':
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1493 if (p[1] == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1494 ++p;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1495 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1496 assert(isNewlineEnd(p));
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1497 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1498 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1499 setLineBegin(p);
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1500 return '\n';
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1501 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1502 if (isUnicodeNewline(p))
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1503 {
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1504 ++p; ++p;
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1505 goto case '\n';
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1506 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1507 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1508 return 0;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1509 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1510 // Skip leading newlines:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1511 while (scanNewline() != 0)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1512 {}
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1513 assert(!isNewline(p));
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1514
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1515 char* begin = p;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1516 c = *p;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1517 closing_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1518 // TODO: Check for non-printable characters?
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1519 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1520 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1521 closing_delim = decodeUTF8();
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1522 if (!isUniAlpha(closing_delim))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1523 break; // Not an identifier.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1524 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1525 else if (!isidbeg(c))
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1526 break; // Not an identifier.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1527
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1528 // Parse Identifier + EndOfLine
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1529 do
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1530 { c = *++p; }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1531 while (isident(c) \|\| !isascii(c) && isUniAlpha(decodeUTF8()))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1532 // Store identifier
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1533 str_delim = begin[0..p-begin];
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1534 // Scan newline
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1535 if (scanNewline() == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1536 --p; // Go back one because of "c = *++p;" in main loop.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1537 else
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1538 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1539 // TODO: error(p, MID.ExpectedNewlineAfterIdentDelim);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1540 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1541 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1542
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1543 bool checkStringDelim(char* p)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1544 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1545 assert(str_delim.length != 0);
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1546 if (buffer[$-1] == '\n' && // Last character copied to buffer must be '\n'.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1547 end-p >= str_delim.length && // Check remaining length.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1548 p[0..str_delim.length] == str_delim) // Compare.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1549 return true;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1550 return false;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1551 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1552
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1553 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1554 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1555 c = *++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1556 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1557 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1558 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1559 if (p[1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1560 ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1561 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1562 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1563 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1564 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1565 setLineBegin(p+1);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1566 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1567 case 0, _Z_:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1568 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedDelimitedString);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1569 goto Lreturn3;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1570 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1571 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1572 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1573 auto begin = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1574 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1575 if (isUnicodeNewlineChar(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1576 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1577 if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1578 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1579 if (str_delim.length)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1580 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1581 if (checkStringDelim(begin))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1582 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1583 p = begin + str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1584 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1585 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1586 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1587 else
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1588 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1589 assert(level == 1);
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1590 --level;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1591 goto Lreturn;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1592 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1593 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1594 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1595 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1596 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1597 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1598 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1599 if (c == opening_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1600 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1601 else if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1602 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1603 if (str_delim.length)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1604 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1605 if (checkStringDelim(p))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1606 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1607 p += str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1608 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1609 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1610 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1611 else if (--level == 0)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1612 goto Lreturn;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1613 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1614 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1615 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1616 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1617 buffer ~= c;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1618 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1619 Lreturn: // Character delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1620 assert(c == closing_delim);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1621 assert(level == 0);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1622 ++p; // Skip closing delimiter.
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1623 Lreturn2: // String delimiter.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1624 if (*p == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1625 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1626 else
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1627 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1628 // TODO: error(p, MID.ExpectedDblQuoteAfterDelim, str_delim.length ? str_delim : closing_delim~"");
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1629 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1630
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1631 t.pf = scanPostfix();
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1632 Lreturn3: // Error.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1633 t.str = buffer ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1634 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1635 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1636
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1637 void scanTokenStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1638 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1639 assert(p[0] == 'q' && p[1] == '{');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1640 t.type = TOK.String;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1641
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1642 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1643 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1644
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1645 // A guard against changes to particular members:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1646 // this.lineNum_hline and this.errorPath
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1647 ++inTokenString;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1648
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1649 uint lineNum = this.lineNum;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1650 uint level = 1;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1651
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1652 ++p; ++p; // Skip q{
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1653
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1654 auto prev_t = &t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1655 Token* token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1656 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1657 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1658 token = new Token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1659 scan(*token);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1660 // Save the tokens in a doubly linked list.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1661 // Could be useful for various tools.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1662 token.prev = prev_t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1663 prev_t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1664 prev_t = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1665 switch (token.type)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1666 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1667 case TOK.LBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1668 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1669 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1670 case TOK.RBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1671 if (--level == 0)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1672 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1673 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1674 t.next = null;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1675 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1676 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1677 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1678 case TOK.EOF:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1679 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedTokenString);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1680 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1681 t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1682 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1683 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1684 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1685 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1686 break; // Exit loop.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1687 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1688
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1689 assert(token.type == TOK.RBrace \|\| token.type == TOK.EOF);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1690 assert(token.type == TOK.RBrace && t.next is null \|\|
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1691 token.type == TOK.EOF && t.next !is null);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1692
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1693 char[] buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1694 // token points to } or EOF
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1695 if (token.type == TOK.EOF)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1696 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1697 t.end = token.start;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1698 buffer = t.srcText[2..$].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1699 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1700 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1701 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1702 // Assign to buffer before scanPostfix().
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1703 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1704 buffer = t.srcText[2..$-1].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1705 t.pf = scanPostfix();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1706 t.end = p; // Assign again because of postfix.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1707 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1708 // Convert newlines to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1709 if (lineNum != this.lineNum)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1710 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1711 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1712 uint i, j;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1713 for (; i < buffer.length; ++i)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1714 switch (buffer[i])
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1715 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1716 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1717 if (buffer[i+1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1718 ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1719 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1720 assert(isNewlineEnd(buffer.ptr + i));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1721 buffer[j++] = '\n'; // Convert Newline to '\n'.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1722 break;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1723 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1724 if (isUnicodeNewline(buffer.ptr + i))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1725 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1726 ++i; ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1727 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1728 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1729 buffer[j++] = buffer[i]; // Copy.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1730 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1731 buffer.length = j; // Adjust length.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1732 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1733 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1734 t.str = buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1735
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1736 --inTokenString;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1737 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1738 } // version(D2)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1739
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1740 dchar scanEscapeSequence()
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1741 out(result)
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1742 { assert(isEncodable(result)); }
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1743 body
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1744 {
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1745 assert(*p == '\\');
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1746
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1747 auto sequenceStart = p; // Used for error reporting.
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1748
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1749 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1750 uint c = char2ev(*p);
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1751 if (c)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1752 {
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1753 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1754 return c;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1755 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1756
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1757 uint digits = 2;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1758
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1759 switch (*p)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1760 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1761 case 'x':
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1762 assert(c == 0);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1763 while (1)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1764 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1765 ++p;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1766 if (ishexad(*p))
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1767 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1768 c *= 16;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1769 if (*p <= '9')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1770 c += *p - '0';
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1771 else if (*p <= 'F')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1772 c += *p - 'A' + 10;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1773 else
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1774 c += *p - 'a' + 10;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1775
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1776 if (!--digits)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1777 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1778 ++p;
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1779 if (isEncodable(c))
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1780 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1781
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1782 error(sequenceStart, MID.InvalidUnicodeEscapeSequence, sequenceStart[0..p-sequenceStart]);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1783 break;
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1784 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1785 continue;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1786 }
432 3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 430 diff changeset	1787
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1788 error(sequenceStart, MID.InsufficientHexDigits);
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1789 break;
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1790 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1791 break;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1792 case 'u':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1793 digits = 4;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1794 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1795 case 'U':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1796 digits = 8;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1797 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1798 default:
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1799 if (isoctal(*p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1800 {
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1801 assert(c == 0);
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1802 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1803 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1804 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1805 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1806 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1807 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1808 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1809 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1810 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1811 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1812 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1813 ++p;
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1814 return c; // Return valid escape value.
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1815 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1816 else if(*p == '&')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1817 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1818 if (isalpha(*++p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1819 {
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1820 auto begin = p;
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1821 while (isalnum(*++p))
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1822 {}
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1823
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1824 if (*p == ';')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1825 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1826 // Pass entity excluding '&' and ';'.
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1827 c = entity2Unicode(begin[0..p - begin]);
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1828 ++p; // Skip ;
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1829 if (c != 0xFFFF)
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1830 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1831 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1832 error(sequenceStart, MID.UndefinedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1833 }
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1834 else
432 3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 430 diff changeset	1835 error(sequenceStart, MID.UnterminatedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1836 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1837 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1838 error(sequenceStart, MID.InvalidBeginHTMLEntity);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1839 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1840 else if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1841 error(sequenceStart, MID.UndefinedEscapeSequence,
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1842 (p == 0 \|\| p == _Z_) ? `\EOF` : `\NewLine`);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1843 else
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1844 {
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1845 char[] str = `\`;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1846 if (isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1847 str ~= *p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1848 else
387 ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1849 encodeUTF8(str, decodeUTF8());
ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1850 ++p;
ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1851 // TODO: check for unprintable character?
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1852 error(sequenceStart, MID.UndefinedEscapeSequence, str);
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1853 }
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1854 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1855 return REPLACEMENT_CHAR; // Error: return replacement character.
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1856 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1857
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1858 /*
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1859 IntegerLiteral:= (Dec\|Hex\|Bin\|Oct)Suffix?
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1860 Dec:= (0\|[1-9][0-9_]*)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1861 Hex:= 0[xX] HexDigits
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1862 Bin:= 0[bB][01_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1863 Oct:= 0[0-7_]+
68 7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	1864 Suffix:= (L[uU]?\|[uU]L?)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1865 HexDigits:= [0-9a-zA-Z_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1866
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1867 Invalid: "0b_", "0x_", "._"
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1868 */
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1869 void scanNumber(ref Token t)
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1870 {
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1871 ulong ulong_;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1872 bool overflow;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1873 bool isDecimal;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1874 size_t digits;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1875
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1876 if (*p != '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1877 goto LscanInteger;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1878 ++p; // skip zero
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1879 // check for xX bB ...
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1880 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1881 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1882 case 'x','X':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1883 goto LscanHex;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1884 case 'b','B':
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1885 goto LscanBinary;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1886 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1887 if (p[1] == 'i')
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1888 goto LscanReal; // 0Li
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1889 break; // 0L
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1890 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1891 if (p[1] == '.')
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1892 break; // 0..
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1893 // 0.
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1894 case 'i','f','F', // Imaginary and float literal suffixes.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1895 'e', 'E': // Float exponent.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1896 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1897 default:
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1898 if (*p == '_')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1899 goto LscanOctal; // 0_
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1900 else if (isdigit(*p))
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1901 {
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1902 if (p == '8' \|\| p == '9')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1903 goto Loctal_hasDecimalDigits; // 08 or 09
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1904 else
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1905 goto Loctal_enter_loop; // 0[0-7]
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1906 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1907 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1908
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1909 // Number 0
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1910 assert(p[-1] == '0');
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1911 assert(p != '_' && !isdigit(p));
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1912 assert(ulong_ == 0);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1913 isDecimal = true;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1914 goto Lfinalize;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1915
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1916 LscanInteger:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1917 assert(p != 0 && isdigit(p));
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1918 isDecimal = true;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1919 goto Lenter_loop_int;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1920 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1921 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1922 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1923 continue;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1924 if (!isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1925 break;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1926 Lenter_loop_int:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1927 if (ulong_ < ulong.max/10 \|\| (ulong_ == ulong.max/10 && *p <= '5'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1928 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1929 ulong_ *= 10;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1930 ulong_ += *p - '0';
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1931 continue;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1932 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1933 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1934 overflow = true;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1935 while (isdigit(*++p)) {}
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1936 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1937 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1938
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1939 // The number could be a float, so check overflow below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1940 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1941 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1942 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1943 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1944 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1945 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1946 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1947 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1948 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1949 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1950 goto LscanReal;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1951 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1952 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1953
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1954 if (overflow)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1955 error(t.start, MID.OverflowDecimalNumber);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1956
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1957 assert((isdigit(p[-1]) \|\| p[-1] == '_') && !isdigit(p) && p != '_');
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1958 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1959
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1960 LscanHex:
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1961 assert(digits == 0);
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	1962 assert(p == 'x' \|\| p == 'X');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1963 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1964 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1965 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1966 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1967 if (!ishexad(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1968 break;
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1969 ++digits;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1970 ulong_ *= 16;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1971 if (*p <= '9')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1972 ulong_ += *p - '0';
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1973 else if (*p <= 'F')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1974 ulong_ += *p - 'A' + 10;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1975 else
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1976 ulong_ += *p - 'a' + 10;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1977 }
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1978
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	1979 assert(ishexad(p[-1]) \|\| p[-1] == '_' \|\| p[-1] == 'x' \|\| p[-1] == 'X');
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1980 assert(!ishexad(p) && p != '_');
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1981
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1982 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1983 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1984 case '.':
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1985 if (p[1] == '.')
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1986 break;
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1987 case 'p', 'P':
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1988 return scanHexReal(t);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1989 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1990 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1991
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1992 if (digits == 0 \|\| digits > 16)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1993 error(t.start, digits == 0 ? MID.NoDigitsInHexNumber : MID.OverflowHexNumber);
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1994
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1995 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1996
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1997 LscanBinary:
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1998 assert(digits == 0);
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	1999 assert(p == 'b' \|\| p == 'B');
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2000 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2001 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2002 if (*++p == '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2003 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2004 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2005 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2006 }
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2007 else if (*p == '1')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2008 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2009 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2010 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2011 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2012 }
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2013 else if (*p == '_')
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2014 continue;
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2015 else
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2016 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2017 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2018
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2019 if (digits == 0 \|\| digits > 64)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2020 error(t.start, digits == 0 ? MID.NoDigitsInBinNumber : MID.OverflowBinaryNumber);
59 3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	2021
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	2022 assert(p[-1] == '0' \|\| p[-1] == '1' \|\| p[-1] == '_' \|\| p[-1] == 'b' \|\| p[-1] == 'B', p[-1] ~ "");
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2023 assert( !(p == '0' \|\| p == '1' \|\| *p == '_') );
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2024 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2025
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2026 LscanOctal:
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2027 assert(*p == '_');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2028 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2029 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2030 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2031 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2032 if (!isoctal(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2033 break;
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2034 Loctal_enter_loop:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2035 if (ulong_ < ulong.max/2 \|\| (ulong_ == ulong.max/2 && *p <= '1'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2036 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2037 ulong_ *= 8;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2038 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2039 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2040 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2041 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2042 overflow = true;
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2043 while (isoctal(*++p)) {}
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2044 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2045 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2046
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2047 bool hasDecimalDigits;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2048 if (isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2049 {
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2050 Loctal_hasDecimalDigits:
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2051 hasDecimalDigits = true;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2052 while (isdigit(*++p)) {}
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2053 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2054
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2055 // The number could be a float, so check errors below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2056 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2057 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2058 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2059 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2060 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2061 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2062 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2063 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2064 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2065 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2066 goto LscanReal;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2067 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2068 }
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2069
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2070 if (hasDecimalDigits)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2071 error(t.start, MID.OctalNumberHasDecimals);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2072
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2073 if (overflow)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2074 error(t.start, MID.OverflowOctalNumber);
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2075 // goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2076
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2077 Lfinalize:
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2078 enum Suffix
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2079 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2080 None = 0,
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2081 Unsigned = 1,
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2082 Long = 2
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2083 }
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2084
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2085 // Scan optional suffix: L, Lu, LU, u, uL, U or UL.
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2086 Suffix suffix;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2087 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2088 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2089 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2090 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2091 case 'L':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2092 if (suffix & Suffix.Long)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2093 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2094 suffix \|= Suffix.Long;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2095 ++p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2096 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2097 case 'u', 'U':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2098 if (suffix & Suffix.Unsigned)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2099 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2100 suffix \|= Suffix.Unsigned;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2101 ++p;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2102 continue;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2103 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2104 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2105 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2106 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2107 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2108
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2109 // Determine type of Integer.
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2110 switch (suffix)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2111 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2112 case Suffix.None:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2113 if (ulong_ & 0x8000_0000_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2114 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2115 if (isDecimal)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2116 error(t.start, MID.OverflowDecimalSign);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2117 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2118 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2119 else if (ulong_ & 0xFFFF_FFFF_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2120 t.type = TOK.Int64;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2121 else if (ulong_ & 0x8000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2122 t.type = isDecimal ? TOK.Int64 : TOK.Uint32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2123 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2124 t.type = TOK.Int32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2125 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2126 case Suffix.Unsigned:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2127 if (ulong_ & 0xFFFF_FFFF_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2128 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2129 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2130 t.type = TOK.Uint32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2131 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2132 case Suffix.Long:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2133 if (ulong_ & 0x8000_0000_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2134 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2135 if (isDecimal)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2136 error(t.start, MID.OverflowDecimalSign);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2137 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2138 }
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2139 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2140 t.type = TOK.Int64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2141 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2142 case Suffix.Unsigned \| Suffix.Long:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2143 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2144 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2145 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2146 assert(0);
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2147 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2148 t.ulong_ = ulong_;
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	2149 t.end = p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2150 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2151 LscanReal:
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2152 scanReal(t);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2153 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2154 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2155
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2156 /*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2157 FloatLiteral:= Float[fFL]?i?
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2158 Float:= DecFloat \| HexFloat
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2159 DecFloat:= ([0-9][0-9_][.][0-9_]DecExponent?) \| [.][0-9][0-9_]DecExponent? \| [0-9][0-9_]DecExponent
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2160 DecExponent:= [eE][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2161 HexFloat:= 0[xX](HexDigits[.]HexDigits \| [.][0-9a-zA-Z]HexDigits? \| HexDigits)HexExponent
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2162 HexExponent:= [pP][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2163 */
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2164 void scanReal(ref Token t)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2165 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2166 if (*p == '.')
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2167 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2168 assert(p[1] != '.');
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2169 // This function was called by scan() or scanNumber().
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2170 while (isdigit(++p) \|\| p == '_') {}
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2171 }
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2172 else
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2173 // This function was called by scanNumber().
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2174 assert(delegate ()
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2175 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2176 switch (*p)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2177 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2178 case 'L':
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2179 if (p[1] != 'i')
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2180 return false;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2181 case 'i', 'f', 'F', 'e', 'E':
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2182 return true;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2183 default:
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2184 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2185 return false;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2186 }()
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2187 );
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2188
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2189 // Scan exponent.
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2190 if (p == 'e' \|\| p == 'E')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2191 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2192 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2193 if (p == '-' \|\| p == '+')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2194 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2195 if (isdigit(*p))
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2196 while (isdigit(++p) \|\| p == '_') {}
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2197 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2198 error(t.start, MID.FloatExpMustStartWithDigit);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2199 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2200
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2201 // Copy whole number and remove underscores from buffer.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2202 char[] buffer = t.start[0..p-t.start].dup;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2203 uint j;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2204 foreach (c; buffer)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2205 if (c != '_')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2206 buffer[j++] = c;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2207 buffer.length = j; // Adjust length.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2208 buffer ~= 0; // Terminate for C functions.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2209
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2210 finalizeFloat(t, buffer);
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2211 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2212
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2213 void scanHexReal(ref Token t)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2214 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2215 assert(p == '.' \|\| p == 'p' \|\| *p == 'P');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2216 MID mid;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2217 if (*p == '.')
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2218 while (ishexad(++p) \|\| p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2219 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2220 // Decimal exponent is required.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2221 if (p != 'p' && p != 'P')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2222 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2223 mid = MID.HexFloatExponentRequired;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2224 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2225 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2226 // Scan exponent
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2227 assert(p == 'p' \|\| p == 'P');
393 fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2228 ++p;
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2229 if (p == '+' \|\| p == '-')
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2230 ++p;
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2231 if (!isdigit(*p))
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2232 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2233 mid = MID.HexFloatExpMustStartWithDigit;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2234 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2235 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2236 while (isdigit(++p) \|\| p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2237 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2238 // Copy whole number and remove underscores from buffer.
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2239 char[] buffer = t.start[0..p-t.start].dup;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2240 uint j;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2241 foreach (c; buffer)
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2242 if (c != '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2243 buffer[j++] = c;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2244 buffer.length = j; // Adjust length.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2245 buffer ~= 0; // Terminate for C functions.
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2246 finalizeFloat(t, buffer);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2247 return;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2248 Lerr:
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2249 t.type = TOK.Float32;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2250 t.end = p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2251 error(t.start, mid);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2252 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2253
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2254 void finalizeFloat(ref Token t, string buffer)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2255 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2256 assert(buffer[$-1] == 0);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2257 // Float number is well-formed. Check suffixes and do conversion.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2258 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2259 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2260 case 'f', 'F':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2261 t.type = TOK.Float32;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2262 t.float_ = strtof(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2263 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2264 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2265 case 'L':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2266 t.type = TOK.Float80;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2267 t.real_ = strtold(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2268 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2269 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2270 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2271 t.type = TOK.Float64;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2272 t.double_ = strtod(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2273 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2274 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2275 if (*p == 'i')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2276 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2277 ++p;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2278 t.type += 3; // Switch to imaginary counterpart.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2279 }
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	2280 if (errno() == ERANGE)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2281 error(t.start, MID.OverflowFloatNumber);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2282 t.end = p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2283 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2284
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2285 /// Scan special token: #line Integer [Filespec] EndOfLine
360 b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence(). aziz parents: 350 diff changeset	2286 void scanSpecialTokenSequence(ref Token t)
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2287 {
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	2288 assert(*p == '#');
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2289 t.type = TOK.HashLine;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2290
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2291 MID mid;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2292 auto errorAtColumn = p;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2293
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2294 ++p;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2295 if (p[0] != 'l' \|\| p[1] != 'i' \|\| p[2] != 'n' \|\| p[3] != 'e')
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2296 {
370 ae4afb66768f - Renamed findModule() to findModulePath(). aziz parents: 367 diff changeset	2297 mid = MID.ExpectedIdentifierSTLine;
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2298 goto Lerr;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2299 }
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2300 p += 3;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2301
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2302 // TODO: #line58"path/file" is legal. Require spaces?
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2303 // State.Space could be used for that purpose.
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2304 enum State
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2305 { /+Space,+/ Integer, Filespec, End }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2306
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2307 State state = State.Integer;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2308
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2309 while (!isEndOfLine(++p))
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2310 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2311 if (isspace(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2312 continue;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2313 if (state == State.Integer)
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2314 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2315 if (!isdigit(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2316 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2317 errorAtColumn = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2318 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2319 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2320 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2321 t.tokLineNum = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2322 scan(*t.tokLineNum);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2323 if (t.tokLineNum.type != TOK.Int32 && t.tokLineNum.type != TOK.Uint32)
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2324 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2325 errorAtColumn = t.tokLineNum.start;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2326 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2327 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2328 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2329 --p; // Go one back because scan() advanced p past the integer.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2330 state = State.Filespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2331 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2332 else if (state == State.Filespec)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2333 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2334 if (*p != '"')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2335 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2336 errorAtColumn = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2337 mid = MID.ExpectedFilespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2338 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2339 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2340 t.tokLineFilespec = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2341 t.tokLineFilespec.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2342 t.tokLineFilespec.type = TOK.Filespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2343 while (*++p != '"')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2344 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2345 if (isEndOfLine(p))
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2346 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2347 errorAtColumn = t.tokLineFilespec.start;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2348 mid = MID.UnterminatedFilespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2349 t.tokLineFilespec.end = p;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2350 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2351 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2352 isascii(*p) \|\| decodeUTF8();
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2353 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2354 auto start = t.tokLineFilespec.start +1; // +1 skips '"'
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2355 t.tokLineFilespec.str = start[0 .. p - start];
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2356 t.tokLineFilespec.end = p + 1;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2357 state = State.End;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2358 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2359 else/+ if (state == State.End)+/
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2360 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2361 mid = MID.UnterminatedSpecialToken;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2362 goto Lerr;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2363 }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2364 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2365 assert(isEndOfLine(p));
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2366
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2367 if (state == State.Integer)
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2368 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2369 errorAtColumn = p;
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2370 mid = MID.ExpectedIntegerAfterSTLine;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2371 goto Lerr;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2372 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2373
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2374 // Evaluate #line only when not in token string.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2375 if (!inTokenString && t.tokLineNum)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2376 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2377 this.lineNum_hline = this.lineNum - t.tokLineNum.uint_ + 1;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2378 if (t.tokLineFilespec)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2379 this.errorPath = t.tokLineFilespec.str;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2380 }
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2381 t.end = p;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2382
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2383 return;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2384 Lerr:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2385 t.end = p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2386 error(errorAtColumn, mid);
371 01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	2387 }
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	2388
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2389 /++
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2390 Insert an empty dummy token before t.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2391 Useful in the parsing phase for representing a node in the AST
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2392 that doesn't consume an actual token from the source text.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2393 +/
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2394 Token* insertEmptyTokenBefore(Token* t)
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2395 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2396 assert(t !is null && t.prev !is null);
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2397 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type));
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2398 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type));
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2399
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2400 auto prev_t = t.prev;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2401 auto new_t = new Token;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2402 new_t.type = TOK.Empty;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2403 new_t.start = new_t.end = prev_t.end;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2404 // Link in new token.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2405 prev_t.next = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2406 new_t.prev = prev_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2407 new_t.next = t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2408 t.prev = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2409 return new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2410 }
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	2411
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2412 uint errorLineNumber(uint lineNum)
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	2413 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2414 return lineNum - this.lineNum_hline;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	2415 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	2416
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2417 void error(char* columnPos, MID mid, ...)
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2418 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2419 error_(this.lineNum, this.lineBegin, columnPos, mid, _arguments, _argptr);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2420 }
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2421
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2422 void error(uint lineNum, char* lineBegin, char* columnPos, MID mid, ...)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2423 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2424 error_(lineNum, lineBegin, columnPos, mid, _arguments, _argptr);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2425 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2426
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2427 void error_(uint lineNum, char* lineBegin, char* columnPos, MID mid,
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2428 TypeInfo[] _arguments, void* _argptr)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2429 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2430 lineNum = this.errorLineNumber(lineNum);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2431 auto location = new Location(errorPath, lineNum, lineBegin, columnPos);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2432 auto msg = Format(_arguments, _argptr, GetMsg(mid));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2433 errors ~= new Information(InfoType.Lexer, mid, location, msg);
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2434 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2435
306 9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2436 Token* getTokens()
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	2437 {
306 9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2438 while (nextToken() != TOK.EOF)
9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2439 {}
9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2440 return head;
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	2441 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2442
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2443 /// Scan the whole text until EOF is encountered.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2444 void scanAll()
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2445 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2446 while (nextToken() != TOK.EOF)
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2447 {}
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2448 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2449
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2450 /// HEAD -> Newline -> First Token
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2451 Token* firstToken()
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2452 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2453 return this.head.next.next;
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2454 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2455
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2456 static void loadKeywords(ref Identifier[string] table)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2457 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2458 foreach(k; keywords)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2459 table[k.str] = k;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2460 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2461
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2462 static bool isNonReservedIdentifier(char[] ident)
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2463 {
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2464 if (ident.length == 0)
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2465 return false;
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2466
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2467 static Identifier[string] reserved_ids_table;
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2468 if (reserved_ids_table is null)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2469 loadKeywords(reserved_ids_table);
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2470
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2471 size_t idx = 1; // Index to the 2nd character in ident.
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2472 dchar isFirstCharUniAlpha()
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2473 {
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2474 idx = 0;
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2475 // NB: decode() could throw an Exception which would be
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2476 // caught by the next try-catch-block.
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2477 return isUniAlpha(std.utf.decode(ident, idx));
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2478 }
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2479
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2480 try
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2481 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2482 if (isidbeg(ident[0]) \|\| !isascii(ident[0]) && isFirstCharUniAlpha())
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2483 {
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2484 foreach (dchar c; ident[idx..$])
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2485 if (!isident(c) && !isUniAlpha(c))
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2486 return false;
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2487 }
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2488 }
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2489 catch (Exception)
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2490 return false;
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2491
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2492 return !(ident in reserved_ids_table);
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2493 }
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2494
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2495 /++
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2496 Returns true if d can be encoded as a UTF-8 sequence.
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2497 +/
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2498 bool isEncodable(dchar d)
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2499 {
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2500 return d < 0xD800 \|\|
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2501 (d > 0xDFFF && d <= 0x10FFFF);
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2502 }
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2503
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2504 /++
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2505 There are a total of 66 noncharacters.
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2506 Returns true if this is one of them.
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2507 See_also: Chapter 16.7 Noncharacters in Unicode 5.0
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2508 +/
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2509 bool isNoncharacter(dchar d)
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2510 {
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2511 return 0xFDD0 <= d && d <= 0xFDEF \|\| // 32
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2512 d <= 0x10FFFF && (d & 0xFFFF) >= 0xFFFE; // 34
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2513 }
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2514
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2515 /++
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2516 Returns true if this character is not a noncharacter, not a surrogate
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2517 code point and not higher than 0x10FFFF.
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2518 +/
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2519 bool isValidDecodedChar(dchar d)
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2520 {
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2521 return d < 0xD800 \|\|
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2522 (d > 0xDFFF && d < 0xFDD0) \|\|
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2523 (d > 0xFDEF && d <= 0x10FFFF && (d & 0xFFFF) < 0xFFFE);
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2524 }
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2525
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2526 /// Is this a trail byte of a UTF-8 sequence?
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2527 bool isTrailByte(ubyte b)
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2528 {
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2529 return (b & 0xC0) == 0x80; // 10xx_xxxx
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2530 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2531
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2532 /// Is this a lead byte of a UTF-8 sequence?
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2533 bool isLeadByte(ubyte b)
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2534 {
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2535 return (b & 0xC0) == 0xC0; // 11xx_xxxx
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2536 }
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2537
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2538 dchar decodeUTF8()
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2539 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2540 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2541 char* p = this.p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2542 dchar d = *p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2543
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2544 ++p; // Move to second byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2545 // Error if second byte is not a trail byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2546 if (!isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2547 goto Lerr2;
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2548
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2549 // Check for overlong sequences.
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2550 switch (d)
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2551 {
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2552 case 0xE0, // 11100000 100xxxxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2553 0xF0, // 11110000 1000xxxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2554 0xF8, // 11111000 10000xxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2555 0xFC: // 11111100 100000xx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2556 if ((*p & d) == 0x80)
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2557 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2558 default:
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2559 if ((d & 0xFE) == 0xC0) // 1100000x
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2560 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2561 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2562
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2563 const char[] checkNextByte = "if (!isTrailByte(*++p))"
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2564 " goto Lerr2;";
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2565 const char[] appendSixBits = "d = (d << 6) \| *p & 0b0011_1111;";
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2566
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2567 // Decode
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2568 if ((d & 0b1110_0000) == 0b1100_0000)
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2569 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2570 // 110xxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2571 d &= 0b0001_1111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2572 mixin(appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2573 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2574 else if ((d & 0b1111_0000) == 0b1110_0000)
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2575 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2576 // 1110xxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2577 d &= 0b0000_1111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2578 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2579 checkNextByte ~ appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2580 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2581 else if ((d & 0b1111_1000) == 0b1111_0000)
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2582 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2583 // 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2584 d &= 0b0000_0111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2585 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2586 checkNextByte ~ appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2587 checkNextByte ~ appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2588 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2589 else
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2590 // 5 and 6 byte UTF-8 sequences are not allowed yet.
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2591 // 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2592 // 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2593 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2594
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2595 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2596
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2597 if (!isEncodable(d))
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2598 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2599 Lerr:
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2600 // Three cases:
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2601 // *) the UTF-8 sequence was successfully decoded but the resulting
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2602 // character is invalid.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2603 // p points to last trail byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2604 // *) the UTF-8 sequence is overlong.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2605 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2606 // *) the UTF-8 sequence has more than 4 bytes or starts with
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2607 // a trail byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2608 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2609 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2610 // Move to next ASCII character or lead byte of a UTF-8 sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2611 while (p < (end-1) && isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2612 ++p;
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2613 --p;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2614 assert(!isTrailByte(p[1]));
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2615 Lerr2:
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2616 d = REPLACEMENT_CHAR;
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2617 error(this.p, MID.InvalidUTF8Sequence);
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2618 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2619
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2620 this.p = p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2621 return d;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2622 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2623
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2624 private void encodeUTF8(ref char[] str, dchar d)
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2625 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2626 char[6] b;
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2627 assert(!isascii(d), "check for ASCII char before calling encodeUTF8().");
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2628 assert(isEncodable(d), "check that 'd' is encodable before calling encodeUTF8().");
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2629
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2630 if (d < 0x800)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2631 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2632 b[0] = 0xC0 \| (d >> 6);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2633 b[1] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2634 str ~= b[0..2];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2635 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2636 else if (d < 0x10000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2637 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2638 b[0] = 0xE0 \| (d >> 12);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2639 b[1] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2640 b[2] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2641 str ~= b[0..3];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2642 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2643 else if (d < 0x200000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2644 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2645 b[0] = 0xF0 \| (d >> 18);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2646 b[1] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2647 b[2] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2648 b[3] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2649 str ~= b[0..4];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2650 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2651 /+ // There are no 5 and 6 byte UTF-8 sequences yet.
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2652 else if (d < 0x4000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2653 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2654 b[0] = 0xF8 \| (d >> 24);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2655 b[1] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2656 b[2] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2657 b[3] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2658 b[4] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2659 str ~= b[0..5];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2660 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2661 else if (d < 0x80000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2662 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2663 b[0] = 0xFC \| (d >> 30);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2664 b[1] = 0x80 \| ((d >> 24) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2665 b[2] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2666 b[3] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2667 b[4] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2668 b[5] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2669 str ~= b[0..6];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2670 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2671 +/
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2672 else
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2673 assert(0);
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2674 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	2675 }
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2676
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2677 unittest
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2678 {
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	2679 Stdout("Testing Lexer.\n");
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2680 struct Pair
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2681 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2682 char[] tokenText;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2683 TOK type;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2684 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2685 static Pair[] pairs = [
494 9a7ca8c56e59 Refactored a few things in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	2686 {"#!äöüß", TOK.Shebang}, {"\n", TOK.Newline},
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2687 {"//çay", TOK.Comment}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2688 {"&", TOK.AndBinary},
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2689 {"/çağ/", TOK.Comment}, {"&&", TOK.AndLogical},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2690 {"/+çak+/", TOK.Comment}, {"&=", TOK.AndAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2691 {">", TOK.Greater}, {"+", TOK.Plus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2692 {">=", TOK.GreaterEqual}, {"++", TOK.PlusPlus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2693 {">>", TOK.RShift}, {"+=", TOK.PlusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2694 {">>=", TOK.RShiftAssign}, {"-", TOK.Minus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2695 {">>>", TOK.URShift}, {"--", TOK.MinusMinus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2696 {">>>=", TOK.URShiftAssign}, {"-=", TOK.MinusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2697 {"<", TOK.Less}, {"=", TOK.Assign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2698 {"<=", TOK.LessEqual}, {"==", TOK.Equal},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2699 {"<>", TOK.LorG}, {"~", TOK.Tilde},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2700 {"<>=", TOK.LorEorG}, {"~=", TOK.CatAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2701 {"<<", TOK.LShift}, {"*", TOK.Mul},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2702 {"<<=", TOK.LShiftAssign}, {"*=", TOK.MulAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2703 {"!", TOK.Not}, {"/", TOK.Div},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2704 {"!=", TOK.NotEqual}, {"/=", TOK.DivAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2705 {"!<", TOK.UorGorE}, {"^", TOK.Xor},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2706 {"!>", TOK.UorLorE}, {"^=", TOK.XorAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2707 {"!<=", TOK.UorG}, {"%", TOK.Mod},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2708 {"!>=", TOK.UorL}, {"%=", TOK.ModAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2709 {"!<>", TOK.UorE}, {"(", TOK.LParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2710 {"!<>=", TOK.Unordered}, {")", TOK.RParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2711 {".", TOK.Dot}, {"[", TOK.LBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2712 {"..", TOK.Slice}, {"]", TOK.RBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2713 {"...", TOK.Ellipses}, {"{", TOK.LBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2714 {"\|", TOK.OrBinary}, {"}", TOK.RBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2715 {"\|\|", TOK.OrLogical}, {":", TOK.Colon},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2716 {"\|=", TOK.OrAssign}, {";", TOK.Semicolon},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2717 {"?", TOK.Question}, {",", TOK.Comma},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2718 {"$", TOK.Dollar}, {"cam", TOK.Identifier},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2719 {"çay", TOK.Identifier}, {".0", TOK.Float64},
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2720 {"0", TOK.Int32}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2721 {"\r", TOK.Newline}, {"\r\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2722 {"\u2028", TOK.Newline}, {"\u2029", TOK.Newline}
40 9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2723 ];
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2724
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2725 char[] src;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2726
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2727 // Join all token texts into a single string.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2728 foreach (i, pair; pairs)
494 9a7ca8c56e59 Refactored a few things in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	2729 if (pair.type == TOK.Comment && pair.tokenText[1] == '/' \|\| // Line comment.
9a7ca8c56e59 Refactored a few things in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	2730 pair.type == TOK.Shebang)
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2731 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2732 assert(pairs[i+1].type == TOK.Newline); // Must be followed by a newline.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2733 src ~= pair.tokenText;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2734 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2735 else
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2736 src ~= pair.tokenText ~ " ";
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2737
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2738 auto lx = new Lexer(src, "");
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2739 auto token = lx.getTokens();
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2740
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2741 uint i;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2742 assert(token == lx.head);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2743 assert(token.next.type == TOK.Newline);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2744 token = token.next.next;
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2745 do
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2746 {
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2747 assert(i < pairs.length);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2748 assert(token.srcText == pairs[i].tokenText, Format("Scanned '{0}' but expected '{1}'", token.srcText, pairs[i].tokenText));
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2749 ++i;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2750 token = token.next;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2751 } while (token.type != TOK.EOF)
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	2752 }
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2753
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2754 unittest
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2755 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2756 Stdout("Testing method Lexer.peek()\n");
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2757 string sourceText = "unittest { }";
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2758 auto lx = new Lexer(sourceText, null);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2759
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2760 auto next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2761 lx.peek(next);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2762 assert(next.type == TOK.Newline);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2763 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2764 assert(next.type == TOK.Unittest);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2765 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2766 assert(next.type == TOK.LBrace);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2767 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2768 assert(next.type == TOK.RBrace);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2769 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2770 assert(next.type == TOK.EOF);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2771
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2772 lx = new Lexer("", null);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2773 next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2774 lx.peek(next);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2775 assert(next.type == TOK.Newline);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2776 lx.peek(next);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2777 assert(next.type == TOK.EOF);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2778 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2779
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2780 unittest
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2781 {
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2782 // Numbers unittest
68 7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2783 // 0L 0ULi 0_L 0_UL 0x0U 0x0p2 0_Fi 0_e2 0_F 0_i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2784 // 0u 0U 0uL 0UL 0L 0LU 0Lu
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2785 // 0Li 0f 0F 0fi 0Fi 0i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2786 // 0b_1_LU 0b1000u
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2787 // 0x232Lu
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2788 }
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2789
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2790 /// ASCII character properties table.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2791 static const int ptable[256] = [
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2792 0, 0, 0, 0, 0, 0, 0, 0, 0,32, 0,32,32, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2793 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2794 32, 0, 0x2200, 0, 0, 0, 0, 0x2700, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2795 7, 7, 7, 7, 7, 7, 7, 7, 6, 6, 0, 0, 0, 0, 0, 0x3f00,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2796 0,12,12,12,12,12,12, 8, 8, 8, 8, 8, 8, 8, 8, 8,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2797 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0, 0x5c00, 0, 0,16,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2798 0, 0x70c, 0x80c,12,12,12, 0xc0c, 8, 8, 8, 8, 8, 8, 8, 0xa08, 8,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2799 8, 8, 0xd08, 8, 0x908, 8, 0xb08, 8, 8, 8, 8, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2800 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2801 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2802 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2803 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2804 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2805 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2806 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2807 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2808 ];
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2809
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2810 enum CProperty
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2811 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2812 Octal = 1,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2813 Digit = 1<<1,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2814 Hex = 1<<2,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2815 Alpha = 1<<3,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2816 Underscore = 1<<4,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2817 Whitespace = 1<<5
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2818 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2819
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2820 const uint EVMask = 0xFF00; // Bit mask for escape value
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2821
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2822 private alias CProperty CP;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2823 int isoctal(char c) { return ptable[c] & CP.Octal; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2824 int isdigit(char c) { return ptable[c] & CP.Digit; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2825 int ishexad(char c) { return ptable[c] & CP.Hex; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2826 int isalpha(char c) { return ptable[c] & CP.Alpha; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2827 int isalnum(char c) { return ptable[c] & (CP.Alpha \| CP.Digit); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2828 int isidbeg(char c) { return ptable[c] & (CP.Alpha \| CP.Underscore); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2829 int isident(char c) { return ptable[c] & (CP.Alpha \| CP.Underscore \| CP.Digit); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2830 int isspace(char c) { return ptable[c] & CP.Whitespace; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2831 int char2ev(char c) { return ptable[c] >> 8; /(ptable[c] & EVMask) >> 8;/ }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2832 int isascii(uint c) { return c < 128; }
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2833
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2834 version(gen_ptable)
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2835 static this()
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2836 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2837 alias ptable p;
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2838 assert(p.length == 256);
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2839 // Initialize character properties table.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2840 for (int i; i < p.length; ++i)
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2841 {
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2842 p[i] = 0; // Reset
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2843 if ('0' <= i && i <= '7')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2844 p[i] \|= CP.Octal;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2845 if ('0' <= i && i <= '9')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2846 p[i] \|= CP.Digit;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2847 if (isdigit(i) \|\| 'a' <= i && i <= 'f' \|\| 'A' <= i && i <= 'F')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2848 p[i] \|= CP.Hex;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2849 if ('a' <= i && i <= 'z' \|\| 'A' <= i && i <= 'Z')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2850 p[i] \|= CP.Alpha;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2851 if (i == '_')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2852 p[i] \|= CP.Underscore;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2853 if (i == ' ' \|\| i == '\t' \|\| i == '\v' \|\| i == '\f')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2854 p[i] \|= CP.Whitespace;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2855 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2856 // Store escape sequence values in second byte.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2857 assert(CProperty.max <= ubyte.max, "character property flags and escape value byte overlap.");
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2858 p['\''] \|= 39 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2859 p['"'] \|= 34 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2860 p['?'] \|= 63 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2861 p['\\'] \|= 92 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2862 p['a'] \|= 7 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2863 p['b'] \|= 8 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2864 p['f'] \|= 12 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2865 p['n'] \|= 10 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2866 p['r'] \|= 13 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2867 p['t'] \|= 9 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2868 p['v'] \|= 11 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2869 // Print a formatted array literal.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2870 char[] array = "[\n";
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2871 foreach (i, c; ptable)
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2872 {
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2873 array ~= Format((c>255?" 0x{0:x},":"{0,2},"), c) ~ (((i+1) % 16) ? "":"\n");
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2874 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2875 array[$-2..$] = "\n]";
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2876 Stdout(array).newline;
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2877 }

Mercurial > projects > dil

annotate trunk/src/dil/Lexer.d @ 494:9a7ca8c56e59