projects/dil: trunk/src/dil/lexer/Lexer.d annotate

annotate trunk/src/dil/lexer/Lexer.d @ 601:9f96fd51cb41

Moved dil.IdTable to dil.lexer.IdTable.

author	Aziz K?ksal <aziz.koeksal@gmail.com>
date	Sun, 06 Jan 2008 00:55:37 +0100
parents	041eae272362
children	2ed1e6d638cd

rev	line source
0 8ba2570de175 Initial import. aziz parents: diff changeset	1 /++
8ba2570de175 Initial import. aziz parents: diff changeset	2 Author: Aziz Köksal
249 32d354584b28 - Upgraded license notices to GPL3. aziz parents: 239 diff changeset	3 License: GPL3
0 8ba2570de175 Initial import. aziz parents: diff changeset	4 +/
576 0df647660e76 Moved Lexer.d to new package 'lexer'. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 552 diff changeset	5 module dil.lexer.Lexer;
577 9e811db780a6 Moved LexerFuncs.d to package 'lexer'. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 576 diff changeset	6
596 39fac5531b85 Moved dil.Token to dil.lexer.Token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 577 diff changeset	7 import dil.lexer.Token;
599 c4cdea3a65ba Moved dil.Keywords to dil.lexer.Keywords. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 596 diff changeset	8 import dil.lexer.Keywords;
600 041eae272362 Moved dil.Identifier to dil.lexer.Identifier. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 599 diff changeset	9 import dil.lexer.Identifier;
601 9f96fd51cb41 Moved dil.IdTable to dil.lexer.IdTable. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 600 diff changeset	10 import dil.lexer.IdTable;
327 a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	11 import dil.Information;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	12 import dil.Messages;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	13 import dil.HtmlEntities;
464 325714d8aa6c Added new module with compiler constants to resolve import cycle. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 434 diff changeset	14 import dil.CompilerInfo;
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	15 import dil.Unicode;
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	16 import tango.stdc.stdlib : strtof, strtod, strtold;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	17 import tango.stdc.errno : errno, ERANGE;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	18 import tango.stdc.time : time_t, time, ctime;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	19 import tango.stdc.string : strlen;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	20 import common;
0 8ba2570de175 Initial import. aziz parents: diff changeset	21
577 9e811db780a6 Moved LexerFuncs.d to package 'lexer'. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 576 diff changeset	22 public import dil.lexer.Funcs;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	23
499 52447db67938 Implemented global table of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 498 diff changeset	24 /++
52447db67938 Implemented global table of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 498 diff changeset	25 The Lexer analyzes the characters of a source text and
52447db67938 Implemented global table of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 498 diff changeset	26 produces a doubly-linked list of tokens.
52447db67938 Implemented global table of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 498 diff changeset	27 +/
0 8ba2570de175 Initial import. aziz parents: diff changeset	28 class Lexer
8ba2570de175 Initial import. aziz parents: diff changeset	29 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	30 Token* head; /// The head of the doubly linked token list.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	31 Token* tail; /// The tail of the linked list. Set in scan().
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	32 Token* token; /// Points to the current token in the token list.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	33 string text; /// The source text.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	34 char[] filePath; /// Path to the source text.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	35 char* p; /// Points to the current character in the source text.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	36 char* end; /// Points one character past the end of the source text.
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	37
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	38 // Members used for error messages:
532 50e64bab9c7a Renamed InformationManager to InfoManager. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 520 diff changeset	39 InfoManager infoMan;
514 6ddff941862a Added new error classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 510 diff changeset	40 LexerError[] errors;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	41 /// Always points to the beginning of the current line.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	42 char* lineBegin;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	43 // Token* newline; /// Current newline token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	44 uint lineNum = 1; /// Current, actual source text line number.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	45 uint lineNum_hline; /// Line number set by #line.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	46 uint inTokenString; /// > 0 if inside q{ }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	47 char[] errorPath; /// The path displayed in error messages.
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	48
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	49 /++
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	50 Construct a Lexer object.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	51 Params:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	52 text = the UTF-8 source code.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	53 filePath = the path to the source code; used for error messages.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	54 +/
532 50e64bab9c7a Renamed InformationManager to InfoManager. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 520 diff changeset	55 this(string text, string filePath, InfoManager infoMan = null)
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	56 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	57 this.filePath = this.errorPath = filePath;
520 f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	58 this.infoMan = infoMan;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	59
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	60 this.text = text;
397 c99f8aeb7b4a Empty source files are handled correctly now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 396 diff changeset	61 if (text.length == 0 \|\| text[$-1] != 0)
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	62 {
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	63 this.text.length = this.text.length + 1;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	64 this.text[$-1] = 0;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	65 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	66
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	67 this.p = this.text.ptr;
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	68 this.end = this.p + this.text.length;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	69 this.lineBegin = this.p;
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	70
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	71 this.head = new Token;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	72 this.head.type = TOK.HEAD;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	73 this.head.start = this.head.end = this.p;
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	74 this.token = this.head;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	75 // Add a newline as the first token after the head.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	76 auto newline = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	77 newline.type = TOK.Newline;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	78 newline.setWhitespaceFlag();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	79 newline.start = newline.end = this.p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	80 newline.filePath = this.errorPath;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	81 newline.lineNum = 1;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	82 newline.lineNum_hline = 0;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	83 // Link in.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	84 this.token.next = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	85 newline.prev = this.token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	86 this.token = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	87 // this.newline = newline;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	88 scanShebang();
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	89 }
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	90
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	91 ~this()
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	92 {
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	93 auto token = head.next;
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	94 while (token !is null)
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	95 {
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	96 assert(token.type == TOK.EOF ? token == tail && token.next is null : 1);
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	97 delete token.prev;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	98 token = token.next;
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	99 }
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	100 delete tail;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	101 }
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	102
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	103 /++
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	104 The "shebang" may optionally appear once at the beginning of a file.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	105 Regexp: #![^\EndOfLine]*
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	106 +/
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	107 void scanShebang()
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	108 {
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	109 if (*p == '#' && p[1] == '!')
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	110 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	111 auto t = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	112 t.type = TOK.Shebang;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	113 t.setWhitespaceFlag();
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	114 t.start = p;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	115 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	116 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	117 isascii(*p) \|\| decodeUTF8();
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	118 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	119 this.token.next = t;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	120 t.prev = this.token;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	121 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	122 }
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	123
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	124 void finalizeSpecialToken(ref Token t)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	125 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	126 assert(t.srcText[0..2] == "__");
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	127 switch (t.type)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	128 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	129 case TOK.FILE:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	130 t.str = this.errorPath;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	131 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	132 case TOK.LINE:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	133 t.uint_ = this.errorLineNumber(this.lineNum);
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	134 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	135 case TOK.DATE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	136 TOK.TIME,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	137 TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	138 time_t time_val;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	139 time(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	140 char* str = ctime(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	141 char[] time_str = str[0 .. strlen(str)];
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	142 switch (t.type)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	143 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	144 case TOK.DATE:
346 ce9e2e77743f - Fix: include one space when slicing time_str for __DATE__. aziz parents: 344 diff changeset	145 time_str = time_str[4..11] ~ time_str[20..24] ~ \0; break;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	146 case TOK.TIME:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	147 time_str = time_str[11..19] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	148 case TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	149 time_str = time_str[0..24] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	150 default: assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	151 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	152 t.str = time_str;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	153 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	154 case TOK.VENDOR:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	155 t.str = VENDOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	156 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	157 case TOK.VERSION:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	158 t.uint_ = VERSION_MAJOR*1000 + VERSION_MINOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	159 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	160 default:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	161 assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	162 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	163 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	164
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	165 private void setLineBegin(char* p)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	166 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	167 // Check that we can look behind one character.
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	168 assert((p-1) >= text.ptr && p < end);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	169 // Check that previous character is a newline.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	170 assert(isNewlineEnd(p - 1));
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	171 this.lineBegin = p;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	172 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	173
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	174 private void scanNext(ref Token* t)
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	175 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	176 assert(t !is null);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	177 if (t.next)
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	178 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	179 t = t.next;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	180 // if (t.type == TOK.Newline)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	181 // this.newline = t;
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	182 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	183 else if (t != this.tail)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	184 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	185 Token* new_t = new Token;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	186 scan(*new_t);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	187 new_t.prev = t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	188 t.next = new_t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	189 t = new_t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	190 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	191 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	192
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	193 /// Advance t one token forward.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	194 void peek(ref Token* t)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	195 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	196 scanNext(t);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	197 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	198
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	199 /// Advance to the next token in the source text.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	200 TOK nextToken()
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	201 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	202 scanNext(this.token);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	203 return this.token.type;
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	204 }
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	205
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	206 /// Returns true if p points to the last character of a Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	207 bool isNewlineEnd(char* p)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	208 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	209 if (p == '\n' \|\| p == '\r')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	210 return true;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	211 if (p == LS[2] \|\| p == PS[2])
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	212 if ((p-2) >= text.ptr)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	213 if (p[-1] == LS[1] && p[-2] == LS[0])
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	214 return true;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	215 return false;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	216 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	217
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	218 /++
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	219 This is the old scan method.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	220 TODO: profile old and new to see which one is faster.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	221 +/
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	222 public void scan(ref Token t)
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	223 in
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	224 {
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	225 assert(text.ptr <= p && p < end);
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	226 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	227 out
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	228 {
397 c99f8aeb7b4a Empty source files are handled correctly now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 396 diff changeset	229 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type));
c99f8aeb7b4a Empty source files are handled correctly now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 396 diff changeset	230 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type));
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	231 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	232 body
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	233 {
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	234 // Scan whitespace.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	235 if (isspace(*p))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	236 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	237 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	238 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	239 {}
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	240 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	241
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	242 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	243 uint c = *p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	244 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	245 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	246 // Newline.
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	247 switch (*p)
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	248 {
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	249 case '\r':
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	250 if (p[1] == '\n')
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	251 ++p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	252 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	253 assert(isNewlineEnd(p));
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	254 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	255 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	256 setLineBegin(p);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	257 // this.newline = &t;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	258 t.type = TOK.Newline;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	259 t.setWhitespaceFlag();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	260 t.filePath = this.errorPath;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	261 t.lineNum = lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	262 t.lineNum_hline = lineNum_hline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	263 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	264 return;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	265 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	266 if (isUnicodeNewline(p))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	267 {
412 fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	268 ++p; ++p;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	269 goto case '\n';
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	270 }
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	271 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	272 // Identifier or string literal.
13 e5211758b63c - Added isidbeg() function. aziz parents: 12 diff changeset	273 if (isidbeg(c))
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	274 {
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	275 if (c == 'r' && p[1] == '"' && ++p)
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	276 return scanRawStringLiteral(t);
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	277 if (c == 'x' && p[1] == '"')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	278 return scanHexStringLiteral(t);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	279 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	280 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	281 if (c == 'q' && p[1] == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	282 return scanDelimitedStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	283 if (c == 'q' && p[1] == '{')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	284 return scanTokenStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	285 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	286 // Scan identifier.
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	287 Lidentifier:
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	288 do
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	289 { c = *++p; }
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	290 while (isident(c) \|\| !isascii(c) && isUnicodeAlpha())
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	291
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	292 t.end = p;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	293
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 499 diff changeset	294 auto id = IdTable.lookup(t.srcText);
499 52447db67938 Implemented global table of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 498 diff changeset	295 t.type = id.type;
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 499 diff changeset	296 t.ident = id;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	297
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	298 if (t.type == TOK.Identifier \|\| t.isKeyword)
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	299 return;
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	300 else if (t.isSpecialToken)
0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	301 finalizeSpecialToken(t);
0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	302 else if (t.type == TOK.EOF)
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	303 {
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	304 tail = &t;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	305 assert(t.srcText == "__EOF__");
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	306 }
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	307 else
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 499 diff changeset	308 assert(0, "unexpected token type: " ~ Token.toString(t.type));
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	309 return;
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	310 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	311
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	312 if (isdigit(c))
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	313 return scanNumber(t);
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	314
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	315 if (c == '/')
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	316 {
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	317 c = *++p;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	318 switch(c)
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	319 {
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	320 case '=':
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	321 ++p;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	322 t.type = TOK.DivAssign;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	323 t.end = p;
cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	324 return;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	325 case '+':
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	326 return scanNestedComment(t);
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	327 case '*':
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	328 return scanBlockComment(t);
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	329 case '/':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	330 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	331 isascii(*p) \|\| decodeUTF8();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	332 t.type = TOK.Comment;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	333 t.setWhitespaceFlag();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	334 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	335 return;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	336 default:
d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	337 t.type = TOK.Div;
10 3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	338 t.end = p;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	339 return;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	340 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	341 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	342
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	343 switch (c)
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	344 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	345 case '\'':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	346 return scanCharacterLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	347 case '`':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	348 return scanRawStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	349 case '"':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	350 return scanNormalStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	351 case '\\':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	352 char[] buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	353 do
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	354 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	355 c = scanEscapeSequence();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	356 if (isascii(c))
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	357 buffer ~= c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	358 else
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	359 encodeUTF8(buffer, c);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	360 } while (*p == '\\')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	361 buffer ~= 0;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	362 t.type = TOK.String;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	363 t.str = buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	364 t.end = p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	365 return;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	366 case '>': /* > >= >> >>= >>> >>>= */
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	367 c = *++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	368 switch (c)
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	369 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	370 case '=':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	371 t.type = TOK.GreaterEqual;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	372 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	373 case '>':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	374 if (p[1] == '>')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	375 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	376 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	377 if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	378 { ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	379 t.type = TOK.URShiftAssign;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	380 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	381 else
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	382 t.type = TOK.URShift;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	383 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	384 else if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	385 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	386 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	387 t.type = TOK.RShiftAssign;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	388 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	389 else
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	390 t.type = TOK.RShift;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	391 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	392 default:
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	393 t.type = TOK.Greater;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	394 goto Lcommon2;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	395 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	396 assert(0);
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	397 case '<': /* < <= <> <>= << <<= */
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	398 c = *++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	399 switch (c)
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	400 {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	401 case '=':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	402 t.type = TOK.LessEqual;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	403 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	404 case '<':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	405 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	406 ++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	407 t.type = TOK.LShiftAssign;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	408 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	409 else
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	410 t.type = TOK.LShift;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	411 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	412 case '>':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	413 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	414 ++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	415 t.type = TOK.LorEorG;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	416 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	417 else
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	418 t.type = TOK.LorG;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	419 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	420 default:
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	421 t.type = TOK.Less;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	422 goto Lcommon2;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	423 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	424 assert(0);
37 7f3bcb97d017 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 36 diff changeset	425 case '!': /* ! !< !> !<= !>= !<> !<>= */
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	426 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	427 switch (c)
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	428 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	429 case '<':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	430 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	431 if (c == '>')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	432 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	433 if (p[1] == '=') {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	434 ++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	435 t.type = TOK.Unordered;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	436 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	437 else
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	438 t.type = TOK.UorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	439 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	440 else if (c == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	441 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	442 t.type = TOK.UorG;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	443 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	444 else {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	445 t.type = TOK.UorGorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	446 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	447 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	448 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	449 case '>':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	450 if (p[1] == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	451 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	452 ++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	453 t.type = TOK.UorL;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	454 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	455 else
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	456 t.type = TOK.UorLorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	457 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	458 case '=':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	459 t.type = TOK.NotEqual;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	460 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	461 default:
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	462 t.type = TOK.Not;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	463 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	464 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	465 assert(0);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	466 case '.': /* . .[0-9] .. ... */
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	467 if (p[1] == '.')
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	468 {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	469 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	470 if (p[1] == '.') {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	471 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	472 t.type = TOK.Ellipses;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	473 }
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	474 else
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	475 t.type = TOK.Slice;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	476 }
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	477 else if (isdigit(p[1]))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	478 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	479 return scanReal(t);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	480 }
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	481 else
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	482 t.type = TOK.Dot;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	483 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	484 case '\|': /* \| \|\| \|= */
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	485 c = *++p;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	486 if (c == '=')
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	487 t.type = TOK.OrAssign;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	488 else if (c == '\|')
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	489 t.type = TOK.OrLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	490 else {
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	491 t.type = TOK.OrBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	492 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	493 }
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	494 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	495 case '&': /* & && &= */
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	496 c = *++p;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	497 if (c == '=')
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	498 t.type = TOK.AndAssign;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	499 else if (c == '&')
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	500 t.type = TOK.AndLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	501 else {
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	502 t.type = TOK.AndBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	503 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	504 }
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	505 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	506 case '+': /* + ++ += */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	507 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	508 if (c == '=')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	509 t.type = TOK.PlusAssign;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	510 else if (c == '+')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	511 t.type = TOK.PlusPlus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	512 else {
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	513 t.type = TOK.Plus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	514 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	515 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	516 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	517 case '-': /* - -- -= */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	518 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	519 if (c == '=')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	520 t.type = TOK.MinusAssign;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	521 else if (c == '-')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	522 t.type = TOK.MinusMinus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	523 else {
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	524 t.type = TOK.Minus;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	525 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	526 }
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	527 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	528 case '=': /* = == */
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	529 if (p[1] == '=') {
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	530 ++p;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	531 t.type = TOK.Equal;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	532 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	533 else
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	534 t.type = TOK.Assign;
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	535 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	536 case '~': /* ~ ~= */
27 43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	537 if (p[1] == '=') {
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	538 ++p;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	539 t.type = TOK.CatAssign;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	540 }
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	541 else
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	542 t.type = TOK.Tilde;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	543 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	544 case '': / * = /
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	545 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	546 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	547 t.type = TOK.MulAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	548 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	549 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	550 t.type = TOK.Mul;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	551 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	552 case '^': /* ^ ^= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	553 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	554 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	555 t.type = TOK.XorAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	556 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	557 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	558 t.type = TOK.Xor;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	559 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	560 case '%': /* % %= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	561 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	562 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	563 t.type = TOK.ModAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	564 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	565 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	566 t.type = TOK.Mod;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	567 goto Lcommon;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	568 // Single character tokens:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	569 case '(':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	570 t.type = TOK.LParen;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	571 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	572 case ')':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	573 t.type = TOK.RParen;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	574 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	575 case '[':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	576 t.type = TOK.LBracket;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	577 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	578 case ']':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	579 t.type = TOK.RBracket;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	580 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	581 case '{':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	582 t.type = TOK.LBrace;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	583 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	584 case '}':
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	585 t.type = TOK.RBrace;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	586 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	587 case ':':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	588 t.type = TOK.Colon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	589 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	590 case ';':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	591 t.type = TOK.Semicolon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	592 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	593 case '?':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	594 t.type = TOK.Question;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	595 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	596 case ',':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	597 t.type = TOK.Comma;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	598 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	599 case '$':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	600 t.type = TOK.Dollar;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	601 Lcommon:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	602 ++p;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	603 Lcommon2:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	604 t.end = p;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	605 return;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	606 case '#':
360 b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence(). aziz parents: 350 diff changeset	607 return scanSpecialTokenSequence(t);
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	608 default:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	609 }
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	610
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	611 // Check for EOF
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	612 if (isEOF(c))
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	613 {
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	614 assert(isEOF(p), ""~p);
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	615 t.type = TOK.EOF;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	616 t.end = p;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	617 tail = &t;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	618 assert(t.start == t.end);
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	619 return;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	620 }
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	621
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	622 if (!isascii(c))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	623 {
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	624 c = decodeUTF8();
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	625 if (isUniAlpha(c))
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	626 goto Lidentifier;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	627 }
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	628
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	629 error(t.start, MID.IllegalCharacter, cast(dchar)c);
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	630
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	631 ++p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	632 t.type = TOK.Illegal;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	633 t.setWhitespaceFlag();
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	634 t.dchar_ = c;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	635 t.end = p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	636 return;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	637 }
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	638 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	639
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	640 template toUint(char[] T)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	641 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	642 static assert(0 < T.length && T.length <= 4);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	643 static if (T.length == 1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	644 const uint toUint = T[0];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	645 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	646 const uint toUint = (T[0] << ((T.length-1)*8)) \| toUint!(T[1..$]);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	647 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	648 static assert(toUint!("\xAA\xBB\xCC\xDD") == 0xAABBCCDD);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	649
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	650 // Can't use this yet due to a bug in DMD (bug id=1534).
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	651 template case_(char[] str, TOK tok, char[] label)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	652 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	653 const char[] case_ =
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	654 `case `~toUint!(str).stringof~`:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	655
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	656 goto `~label~`;`;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	657 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	658
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	659 template case_L4(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	660 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	661 const char[] case_L4 = case_!(str, tok, "Lcommon_4");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	662 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	663
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	664 template case_L3(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	665 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	666 const char[] case_L3 = case_!(str, tok, "Lcommon_3");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	667 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	668
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	669 template case_L2(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	670 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	671 const char[] case_L2 = case_!(str, tok, "Lcommon_2");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	672 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	673
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	674 template case_L1(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	675 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	676 const char[] case_L3 = case_!(str, tok, "Lcommon");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	677 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	678
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	679 public void scan_(ref Token t)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	680 in
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	681 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	682 assert(text.ptr <= p && p < end);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	683 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	684 out
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	685 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	686 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type));
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	687 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type));
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	688 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	689 body
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	690 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	691 // Scan whitespace.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	692 if (isspace(*p))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	693 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	694 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	695 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	696 {}
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	697 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	698
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	699 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	700 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	701 // Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	702 switch (*p)
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	703 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	704 case '\r':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	705 if (p[1] == '\n')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	706 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	707 case '\n':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	708 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	709 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	710 ++lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	711 setLineBegin(p);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	712 // this.newline = &t;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	713 t.type = TOK.Newline;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	714 t.setWhitespaceFlag();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	715 t.filePath = this.errorPath;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	716 t.lineNum = lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	717 t.lineNum_hline = lineNum_hline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	718 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	719 return;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	720 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	721 if (isUnicodeNewline(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	722 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	723 ++p; ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	724 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	725 }
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	726 }
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	727
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	728 uint c = *p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	729 assert(end - p != 0);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	730 switch (end - p)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	731 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	732 case 1:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	733 goto L1character;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	734 case 2:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	735 c <<= 8; c \|= p[1];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	736 goto L2characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	737 case 3:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	738 c <<= 8; c \|= p[1]; c <<= 8; c \|= p[2];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	739 goto L3characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	740 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	741 version(BigEndian)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	742 c = cast(uint)p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	743 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	744 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	745 c <<= 8; c \|= p[1]; c <<= 8; c \|= p[2]; c <<= 8; c \|= p[3];
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	746 /+
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	747 c = cast(uint)p;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	748 asm
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	749 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	750 mov EDX, c;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	751 bswap EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	752 mov c, EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	753 }
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	754 +/
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	755 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	756 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	757
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	758 // 4 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	759 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	760 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	761 case toUint!(">>>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	762 t.type = TOK.RShiftAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	763 goto Lcommon_4;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	764 case toUint!("!<>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	765 t.type = TOK.Unordered;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	766 Lcommon_4:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	767 p += 4;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	768 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	769 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	770 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	771 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	772
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	773 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	774 L3characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	775 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	776 // 3 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	777 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	778 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	779 case toUint!(">>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	780 t.type = TOK.RShiftAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	781 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	782 case toUint!(">>>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	783 t.type = TOK.URShift;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	784 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	785 case toUint!("<>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	786 t.type = TOK.LorEorG;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	787 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	788 case toUint!("<<="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	789 t.type = TOK.LShiftAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	790 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	791 case toUint!("!<="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	792 t.type = TOK.UorG;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	793 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	794 case toUint!("!>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	795 t.type = TOK.UorL;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	796 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	797 case toUint!("!<>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	798 t.type = TOK.UorE;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	799 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	800 case toUint!("..."):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	801 t.type = TOK.Ellipses;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	802 Lcommon_3:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	803 p += 3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	804 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	805 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	806 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	807 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	808
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	809 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	810 L2characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	811 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	812 // 2 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	813 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	814 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	815 case toUint!("/+"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	816 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	817 return scanNestedComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	818 case toUint!("/*"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	819 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	820 return scanBlockComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	821 case toUint!("//"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	822 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	823 assert(*p == '/');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	824 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	825 isascii(*p) \|\| decodeUTF8();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	826 t.type = TOK.Comment;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	827 t.setWhitespaceFlag();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	828 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	829 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	830 case toUint!(">="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	831 t.type = TOK.GreaterEqual;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	832 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	833 case toUint!(">>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	834 t.type = TOK.RShift;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	835 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	836 case toUint!("<<"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	837 t.type = TOK.LShift;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	838 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	839 case toUint!("<="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	840 t.type = TOK.LessEqual;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	841 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	842 case toUint!("<>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	843 t.type = TOK.LorG;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	844 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	845 case toUint!("!<"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	846 t.type = TOK.UorGorE;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	847 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	848 case toUint!("!>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	849 t.type = TOK.UorLorE;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	850 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	851 case toUint!("!="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	852 t.type = TOK.NotEqual;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	853 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	854 case toUint!(".."):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	855 t.type = TOK.Slice;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	856 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	857 case toUint!("&&"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	858 t.type = TOK.AndLogical;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	859 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	860 case toUint!("&="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	861 t.type = TOK.AndAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	862 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	863 case toUint!("\|\|"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	864 t.type = TOK.OrLogical;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	865 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	866 case toUint!("\|="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	867 t.type = TOK.OrAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	868 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	869 case toUint!("++"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	870 t.type = TOK.PlusPlus;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	871 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	872 case toUint!("+="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	873 t.type = TOK.PlusAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	874 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	875 case toUint!("--"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	876 t.type = TOK.MinusMinus;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	877 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	878 case toUint!("-="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	879 t.type = TOK.MinusAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	880 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	881 case toUint!("=="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	882 t.type = TOK.Equal;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	883 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	884 case toUint!("~="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	885 t.type = TOK.CatAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	886 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	887 case toUint!("*="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	888 t.type = TOK.MulAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	889 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	890 case toUint!("/="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	891 t.type = TOK.DivAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	892 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	893 case toUint!("^="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	894 t.type = TOK.XorAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	895 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	896 case toUint!("%="):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	897 t.type = TOK.ModAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	898 Lcommon_2:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	899 p += 2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	900 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	901 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	902 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	903 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	904
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	905 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	906 L1character:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	907 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	908 assert(p == c, Format("p={0},c={1}", p, cast(dchar)c));
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	909 // 1 character tokens.
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	910 // TODO: consider storing the token type in ptable.
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	911 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	912 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	913 case '\'':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	914 return scanCharacterLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	915 case '`':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	916 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	917 case '"':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	918 return scanNormalStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	919 case '\\':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	920 char[] buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	921 do
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	922 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	923 c = scanEscapeSequence();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	924 if (isascii(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	925 buffer ~= c;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	926 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	927 encodeUTF8(buffer, c);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	928 } while (*p == '\\')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	929 buffer ~= 0;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	930 t.type = TOK.String;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	931 t.str = buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	932 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	933 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	934 case '<':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	935 t.type = TOK.Greater;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	936 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	937 case '>':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	938 t.type = TOK.Less;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	939 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	940 case '^':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	941 t.type = TOK.Xor;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	942 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	943 case '!':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	944 t.type = TOK.Not;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	945 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	946 case '.':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	947 if (isdigit(p[1]))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	948 return scanReal(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	949 t.type = TOK.Dot;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	950 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	951 case '&':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	952 t.type = TOK.AndBinary;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	953 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	954 case '\|':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	955 t.type = TOK.OrBinary;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	956 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	957 case '+':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	958 t.type = TOK.Plus;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	959 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	960 case '-':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	961 t.type = TOK.Minus;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	962 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	963 case '=':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	964 t.type = TOK.Assign;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	965 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	966 case '~':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	967 t.type = TOK.Tilde;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	968 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	969 case '*':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	970 t.type = TOK.Mul;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	971 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	972 case '/':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	973 t.type = TOK.Div;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	974 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	975 case '%':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	976 t.type = TOK.Mod;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	977 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	978 case '(':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	979 t.type = TOK.LParen;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	980 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	981 case ')':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	982 t.type = TOK.RParen;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	983 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	984 case '[':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	985 t.type = TOK.LBracket;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	986 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	987 case ']':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	988 t.type = TOK.RBracket;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	989 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	990 case '{':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	991 t.type = TOK.LBrace;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	992 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	993 case '}':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	994 t.type = TOK.RBrace;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	995 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	996 case ':':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	997 t.type = TOK.Colon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	998 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	999 case ';':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1000 t.type = TOK.Semicolon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1001 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1002 case '?':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1003 t.type = TOK.Question;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1004 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1005 case ',':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1006 t.type = TOK.Comma;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1007 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1008 case '$':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1009 t.type = TOK.Dollar;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1010 Lcommon:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1011 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1012 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1013 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1014 case '#':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1015 return scanSpecialTokenSequence(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1016 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1017 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1018
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1019 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1020 assert(*p == c);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1021
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1022 // TODO: consider moving isidbeg() and isdigit() up.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1023 if (isidbeg(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1024 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1025 if (c == 'r' && p[1] == '"' && ++p)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1026 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1027 if (c == 'x' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1028 return scanHexStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1029 version(D2)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1030 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1031 if (c == 'q' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1032 return scanDelimitedStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1033 if (c == 'q' && p[1] == '{')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1034 return scanTokenStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1035 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1036 // Scan identifier.
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1037 Lidentifier:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1038 do
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1039 { c = *++p; }
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	1040 while (isident(c) \|\| !isascii(c) && isUnicodeAlpha())
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1041
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1042 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1043
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 499 diff changeset	1044 auto id = IdTable.lookup(t.srcText);
499 52447db67938 Implemented global table of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 498 diff changeset	1045 t.type = id.type;
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 499 diff changeset	1046 t.ident = id;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1047
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1048 if (t.type == TOK.Identifier \|\| t.isKeyword)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1049 return;
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1050 else if (t.isSpecialToken)
0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1051 finalizeSpecialToken(t);
0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1052 else if (t.type == TOK.EOF)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1053 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1054 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1055 assert(t.srcText == "__EOF__");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1056 }
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1057 else
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 499 diff changeset	1058 assert(0, "unexpected token type: " ~ Token.toString(t.type));
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1059 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1060 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1061
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1062 if (isdigit(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1063 return scanNumber(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1064
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1065 // Check for EOF
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1066 if (isEOF(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1067 {
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1068 assert(isEOF(p), p~"");
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1069 t.type = TOK.EOF;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1070 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1071 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1072 assert(t.start == t.end);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1073 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1074 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1075
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1076 if (!isascii(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1077 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1078 c = decodeUTF8();
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1079 if (isUniAlpha(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1080 goto Lidentifier;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1081 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1082
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1083 error(t.start, MID.IllegalCharacter, cast(dchar)c);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1084
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1085 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1086 t.type = TOK.Illegal;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	1087 t.setWhitespaceFlag();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1088 t.dchar_ = c;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1089 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1090 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1091 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1092
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1093 void scanBlockComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1094 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1095 assert(p[-1] == '/' && p == '');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1096 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1097 auto tokenLineBegin = lineBegin;
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1098 Loop:
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1099 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1100 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1101 switch (*++p)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1102 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1103 case '*':
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1104 if (p[1] != '/')
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1105 continue;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1106 p += 2;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1107 break Loop;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1108 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1109 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1110 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1111 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1112 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1113 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1114 setLineBegin(p+1);
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1115 break;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1116 default:
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1117 if (!isascii(*p))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1118 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1119 if (isUnicodeNewlineChar(decodeUTF8()))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1120 goto case '\n';
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1121 }
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1122 else if (isEOF(*p))
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1123 {
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1124 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedBlockComment);
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1125 break Loop;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1126 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1127 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1128 }
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1129 t.type = TOK.Comment;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	1130 t.setWhitespaceFlag();
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1131 t.end = p;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1132 return;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1133 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1134
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1135 void scanNestedComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1136 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1137 assert(p[-1] == '/' && *p == '+');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1138 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1139 auto tokenLineBegin = lineBegin;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1140 uint level = 1;
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1141 Loop:
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1142 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1143 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1144 switch (*++p)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1145 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1146 case '/':
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1147 if (p[1] == '+')
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1148 ++p, ++level;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1149 continue;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1150 case '+':
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1151 if (p[1] != '/')
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1152 continue;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1153 ++p;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1154 if (--level != 0)
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1155 continue;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1156 ++p;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1157 break Loop;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1158 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1159 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1160 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1161 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1162 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1163 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1164 setLineBegin(p+1);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1165 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1166 default:
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1167 if (!isascii(*p))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1168 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1169 if (isUnicodeNewlineChar(decodeUTF8()))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1170 goto case '\n';
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1171 }
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1172 else if (isEOF(*p))
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1173 {
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1174 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedNestedComment);
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1175 break Loop;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1176 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1177 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1178 }
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1179 t.type = TOK.Comment;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	1180 t.setWhitespaceFlag();
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1181 t.end = p;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1182 return;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1183 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1184
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1185 char scanPostfix()
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1186 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1187 assert(p[-1] == '"' \|\| p[-1] == '`' \|\|
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1188 { version(D2) return p[-1] == '}';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1189 else return 0; }()
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1190 );
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1191 switch (*p)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1192 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1193 case 'c':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1194 case 'w':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1195 case 'd':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1196 return *p++;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1197 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1198 return 0;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1199 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1200 assert(0);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1201 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1202
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1203 void scanNormalStringLiteral(ref Token t)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1204 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1205 assert(*p == '"');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1206 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1207 auto tokenLineBegin = lineBegin;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1208 t.type = TOK.String;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1209 char[] buffer;
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1210 uint c;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1211 while (1)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1212 {
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1213 c = *++p;
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1214 switch (c)
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1215 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1216 case '"':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1217 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1218 t.pf = scanPostfix();
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1219 Lreturn:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1220 t.str = buffer ~ '\0';
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1221 t.end = p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1222 return;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1223 case '\\':
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1224 c = scanEscapeSequence();
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1225 --p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1226 if (isascii(c))
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1227 break;
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1228 encodeUTF8(buffer, c);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1229 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1230 case '\r':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1231 if (p[1] == '\n')
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1232 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1233 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1234 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1235 c = '\n'; // Convert Newline to \n.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1236 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1237 setLineBegin(p+1);
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1238 break;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1239 case 0, _Z_:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1240 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedString);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1241 goto Lreturn;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1242 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1243 if (!isascii(c))
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1244 {
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1245 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1246 if (isUnicodeNewlineChar(c))
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1247 goto case '\n';
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1248 encodeUTF8(buffer, c);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1249 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1250 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1251 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1252 assert(isascii(c));
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1253 buffer ~= c;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1254 }
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1255 assert(0);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1256 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1257
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1258 void scanCharacterLiteral(ref Token t)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1259 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1260 assert(*p == '\'');
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1261 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1262 t.type = TOK.CharLiteral;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1263 switch (*p)
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1264 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1265 case '\\':
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1266 t.dchar_ = scanEscapeSequence();
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1267 break;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1268 case '\'':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1269 error(t.start, MID.EmptyCharacterLiteral);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1270 break;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1271 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1272 if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1273 break;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1274 uint c = *p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1275 if (!isascii(c))
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1276 c = decodeUTF8();
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1277 t.dchar_ = c;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1278 ++p;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1279 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1280
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1281 if (*p == '\'')
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1282 ++p;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1283 else
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1284 error(t.start, MID.UnterminatedCharacterLiteral);
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1285 t.end = p;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1286 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1287
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1288 void scanRawStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1289 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1290 assert(p == '`' \|\| p == '"' && p[-1] == 'r');
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1291 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1292 auto tokenLineBegin = lineBegin;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1293 t.type = TOK.String;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1294 uint delim = *p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1295 char[] buffer;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1296 uint c;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1297 while (1)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1298 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1299 c = *++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1300 switch (c)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1301 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1302 case '\r':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1303 if (p[1] == '\n')
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1304 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1305 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1306 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1307 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1308 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1309 setLineBegin(p+1);
52 f65a83c27638 - Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well. aziz parents: 51 diff changeset	1310 break;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1311 case '`':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1312 case '"':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1313 if (c == delim)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1314 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1315 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1316 t.pf = scanPostfix();
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1317 Lreturn:
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1318 t.str = buffer ~ '\0';
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1319 t.end = p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1320 return;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1321 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1322 break;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1323 case 0, _Z_:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1324 error(tokenLineNum, tokenLineBegin, t.start,
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1325 delim == 'r' ? MID.UnterminatedRawString : MID.UnterminatedBackQuoteString);
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1326 goto Lreturn;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1327 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1328 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1329 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1330 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1331 if (isUnicodeNewlineChar(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1332 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1333 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1334 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1335 }
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1336 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1337 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1338 buffer ~= c;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1339 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1340 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1341 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1342
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1343 void scanHexStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1344 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1345 assert(p[0] == 'x' && p[1] == '"');
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1346 t.type = TOK.String;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1347
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1348 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1349 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1350
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1351 uint c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1352 ubyte[] buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1353 ubyte h; // hex number
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1354 uint n; // number of hex digits
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1355
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1356 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1357 assert(*p == '"');
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1358 while (1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1359 {
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1360 c = *++p;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1361 switch (c)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1362 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1363 case '"':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1364 if (n & 1)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1365 error(tokenLineNum, tokenLineBegin, t.start, MID.OddNumberOfDigitsInHexString);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1366 ++p;
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1367 t.pf = scanPostfix();
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1368 Lreturn:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1369 t.str = cast(string) (buffer ~= 0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1370 t.end = p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1371 return;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1372 case '\r':
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1373 if (p[1] == '\n')
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1374 ++p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1375 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1376 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1377 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1378 setLineBegin(p+1);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1379 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1380 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1381 if (ishexad(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1382 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1383 if (c <= '9')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1384 c -= '0';
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1385 else if (c <= 'F')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1386 c -= 'A' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1387 else
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1388 c -= 'a' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1389
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1390 if (n & 1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1391 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1392 h <<= 4;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1393 h \|= c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1394 buffer ~= h;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1395 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1396 else
112 004d98df65af - Implemented parseInterfaceDeclaration(). aziz parents: 105 diff changeset	1397 h = cast(ubyte)c;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1398 ++n;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1399 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1400 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1401 else if (isspace(c))
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1402 continue; // Skip spaces.
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 494 diff changeset	1403 else if (isEOF(c))
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1404 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1405 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedHexString);
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1406 t.pf = 0;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1407 goto Lreturn;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1408 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1409 else
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1410 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1411 auto errorAt = p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1412 if (!isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1413 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1414 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1415 if (isUnicodeNewlineChar(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1416 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1417 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1418 error(errorAt, MID.NonHexCharInHexString, cast(dchar)c);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1419 }
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1420 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1421 }
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1422 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1423 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1424
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1425 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1426 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1427 void scanDelimitedStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1428 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1429 assert(p[0] == 'q' && p[1] == '"');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1430 t.type = TOK.String;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1431
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1432 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1433 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1434
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1435 char[] buffer;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1436 dchar opening_delim = 0, // 0 if no nested delimiter or '[', '(', '<', '{'
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1437 closing_delim; // Will be ']', ')', '>', '},
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1438 // the first character of an identifier or
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1439 // any other Unicode/ASCII character.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1440 char[] str_delim; // Identifier delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1441 uint level = 1; // Counter for nestable delimiters.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1442
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1443 ++p; ++p; // Skip q"
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1444 uint c = *p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1445 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1446 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1447 case '(':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1448 opening_delim = c;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1449 closing_delim = ')'; // c + 1
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1450 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1451 case '[', '<', '{':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1452 opening_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1453 closing_delim = c + 2; // Get to closing counterpart. Feature of ASCII table.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1454 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1455 default:
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1456 dchar scanNewline()
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1457 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1458 switch (*p)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1459 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1460 case '\r':
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1461 if (p[1] == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1462 ++p;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1463 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1464 assert(isNewlineEnd(p));
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1465 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1466 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1467 setLineBegin(p);
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1468 return '\n';
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1469 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1470 if (isUnicodeNewline(p))
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1471 {
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1472 ++p; ++p;
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1473 goto case '\n';
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1474 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1475 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1476 return 0;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1477 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1478 // Skip leading newlines:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1479 while (scanNewline() != 0)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1480 {}
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1481 assert(!isNewline(p));
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1482
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1483 char* begin = p;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1484 c = *p;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1485 closing_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1486 // TODO: Check for non-printable characters?
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1487 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1488 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1489 closing_delim = decodeUTF8();
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1490 if (!isUniAlpha(closing_delim))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1491 break; // Not an identifier.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1492 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1493 else if (!isidbeg(c))
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1494 break; // Not an identifier.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1495
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1496 // Parse Identifier + EndOfLine
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1497 do
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1498 { c = *++p; }
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	1499 while (isident(c) \|\| !isascii(c) && isUnicodeAlpha())
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1500 // Store identifier
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1501 str_delim = begin[0..p-begin];
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1502 // Scan newline
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1503 if (scanNewline() == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1504 --p; // Go back one because of "c = *++p;" in main loop.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1505 else
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1506 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1507 // TODO: error(p, MID.ExpectedNewlineAfterIdentDelim);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1508 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1509 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1510
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1511 bool checkStringDelim(char* p)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1512 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1513 assert(str_delim.length != 0);
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1514 if (buffer[$-1] == '\n' && // Last character copied to buffer must be '\n'.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1515 end-p >= str_delim.length && // Check remaining length.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1516 p[0..str_delim.length] == str_delim) // Compare.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1517 return true;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1518 return false;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1519 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1520
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1521 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1522 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1523 c = *++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1524 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1525 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1526 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1527 if (p[1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1528 ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1529 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1530 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1531 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1532 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1533 setLineBegin(p+1);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1534 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1535 case 0, _Z_:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1536 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedDelimitedString);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1537 goto Lreturn3;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1538 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1539 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1540 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1541 auto begin = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1542 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1543 if (isUnicodeNewlineChar(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1544 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1545 if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1546 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1547 if (str_delim.length)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1548 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1549 if (checkStringDelim(begin))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1550 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1551 p = begin + str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1552 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1553 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1554 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1555 else
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1556 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1557 assert(level == 1);
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1558 --level;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1559 goto Lreturn;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1560 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1561 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1562 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1563 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1564 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1565 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1566 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1567 if (c == opening_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1568 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1569 else if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1570 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1571 if (str_delim.length)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1572 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1573 if (checkStringDelim(p))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1574 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1575 p += str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1576 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1577 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1578 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1579 else if (--level == 0)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1580 goto Lreturn;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1581 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1582 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1583 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1584 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1585 buffer ~= c;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1586 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1587 Lreturn: // Character delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1588 assert(c == closing_delim);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1589 assert(level == 0);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1590 ++p; // Skip closing delimiter.
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1591 Lreturn2: // String delimiter.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1592 if (*p == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1593 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1594 else
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1595 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1596 // TODO: error(p, MID.ExpectedDblQuoteAfterDelim, str_delim.length ? str_delim : closing_delim~"");
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1597 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1598
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1599 t.pf = scanPostfix();
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1600 Lreturn3: // Error.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1601 t.str = buffer ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1602 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1603 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1604
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1605 void scanTokenStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1606 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1607 assert(p[0] == 'q' && p[1] == '{');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1608 t.type = TOK.String;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1609
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1610 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1611 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1612
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1613 // A guard against changes to particular members:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1614 // this.lineNum_hline and this.errorPath
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1615 ++inTokenString;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1616
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1617 uint lineNum = this.lineNum;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1618 uint level = 1;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1619
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1620 ++p; ++p; // Skip q{
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1621
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1622 auto prev_t = &t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1623 Token* token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1624 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1625 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1626 token = new Token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1627 scan(*token);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1628 // Save the tokens in a doubly linked list.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1629 // Could be useful for various tools.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1630 token.prev = prev_t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1631 prev_t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1632 prev_t = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1633 switch (token.type)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1634 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1635 case TOK.LBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1636 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1637 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1638 case TOK.RBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1639 if (--level == 0)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1640 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1641 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1642 t.next = null;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1643 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1644 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1645 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1646 case TOK.EOF:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1647 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedTokenString);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1648 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1649 t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1650 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1651 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1652 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1653 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1654 break; // Exit loop.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1655 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1656
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1657 assert(token.type == TOK.RBrace \|\| token.type == TOK.EOF);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1658 assert(token.type == TOK.RBrace && t.next is null \|\|
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1659 token.type == TOK.EOF && t.next !is null);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1660
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1661 char[] buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1662 // token points to } or EOF
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1663 if (token.type == TOK.EOF)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1664 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1665 t.end = token.start;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1666 buffer = t.srcText[2..$].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1667 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1668 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1669 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1670 // Assign to buffer before scanPostfix().
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1671 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1672 buffer = t.srcText[2..$-1].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1673 t.pf = scanPostfix();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1674 t.end = p; // Assign again because of postfix.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1675 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1676 // Convert newlines to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1677 if (lineNum != this.lineNum)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1678 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1679 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1680 uint i, j;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1681 for (; i < buffer.length; ++i)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1682 switch (buffer[i])
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1683 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1684 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1685 if (buffer[i+1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1686 ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1687 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1688 assert(isNewlineEnd(buffer.ptr + i));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1689 buffer[j++] = '\n'; // Convert Newline to '\n'.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1690 break;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1691 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1692 if (isUnicodeNewline(buffer.ptr + i))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1693 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1694 ++i; ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1695 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1696 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1697 buffer[j++] = buffer[i]; // Copy.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1698 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1699 buffer.length = j; // Adjust length.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1700 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1701 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1702 t.str = buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1703
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1704 --inTokenString;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1705 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1706 } // version(D2)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1707
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1708 dchar scanEscapeSequence()
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1709 out(result)
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	1710 { assert(isValidChar(result)); }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1711 body
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1712 {
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1713 assert(*p == '\\');
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1714
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1715 auto sequenceStart = p; // Used for error reporting.
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1716
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1717 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1718 uint c = char2ev(*p);
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1719 if (c)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1720 {
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1721 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1722 return c;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1723 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1724
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1725 uint digits = 2;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1726
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1727 switch (*p)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1728 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1729 case 'x':
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1730 assert(c == 0);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1731 while (1)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1732 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1733 ++p;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1734 if (ishexad(*p))
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1735 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1736 c *= 16;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1737 if (*p <= '9')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1738 c += *p - '0';
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1739 else if (*p <= 'F')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1740 c += *p - 'A' + 10;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1741 else
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1742 c += *p - 'a' + 10;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1743
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1744 if (!--digits)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1745 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1746 ++p;
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	1747 if (isValidChar(c))
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1748 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1749
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1750 error(sequenceStart, MID.InvalidUnicodeEscapeSequence, sequenceStart[0..p-sequenceStart]);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1751 break;
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1752 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1753 continue;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1754 }
432 3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 430 diff changeset	1755
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1756 error(sequenceStart, MID.InsufficientHexDigits);
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1757 break;
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1758 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1759 break;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1760 case 'u':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1761 digits = 4;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1762 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1763 case 'U':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1764 digits = 8;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1765 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1766 default:
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1767 if (isoctal(*p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1768 {
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1769 assert(c == 0);
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1770 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1771 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1772 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1773 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1774 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1775 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1776 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1777 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1778 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1779 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1780 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1781 ++p;
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1782 return c; // Return valid escape value.
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1783 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1784 else if(*p == '&')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1785 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1786 if (isalpha(*++p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1787 {
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1788 auto begin = p;
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1789 while (isalnum(*++p))
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1790 {}
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1791
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1792 if (*p == ';')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1793 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1794 // Pass entity excluding '&' and ';'.
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1795 c = entity2Unicode(begin[0..p - begin]);
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1796 ++p; // Skip ;
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1797 if (c != 0xFFFF)
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1798 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1799 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1800 error(sequenceStart, MID.UndefinedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1801 }
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1802 else
432 3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 430 diff changeset	1803 error(sequenceStart, MID.UnterminatedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1804 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1805 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1806 error(sequenceStart, MID.InvalidBeginHTMLEntity);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1807 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1808 else if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1809 error(sequenceStart, MID.UndefinedEscapeSequence,
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 494 diff changeset	1810 isEOF(*p) ? `\EOF` : `\NewLine`);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1811 else
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1812 {
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1813 char[] str = `\`;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1814 if (isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1815 str ~= *p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1816 else
387 ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1817 encodeUTF8(str, decodeUTF8());
ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1818 ++p;
ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1819 // TODO: check for unprintable character?
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1820 error(sequenceStart, MID.UndefinedEscapeSequence, str);
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1821 }
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1822 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1823 return REPLACEMENT_CHAR; // Error: return replacement character.
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1824 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1825
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1826 /*
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1827 IntegerLiteral:= (Dec\|Hex\|Bin\|Oct)Suffix?
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1828 Dec:= (0\|[1-9][0-9_]*)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1829 Hex:= 0[xX] HexDigits
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1830 Bin:= 0[bB][01_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1831 Oct:= 0[0-7_]+
68 7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	1832 Suffix:= (L[uU]?\|[uU]L?)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1833 HexDigits:= [0-9a-zA-Z_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1834
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1835 Invalid: "0b_", "0x_", "._"
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1836 */
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1837 void scanNumber(ref Token t)
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1838 {
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1839 ulong ulong_;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1840 bool overflow;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1841 bool isDecimal;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1842 size_t digits;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1843
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1844 if (*p != '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1845 goto LscanInteger;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1846 ++p; // skip zero
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1847 // check for xX bB ...
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1848 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1849 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1850 case 'x','X':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1851 goto LscanHex;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1852 case 'b','B':
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1853 goto LscanBinary;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1854 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1855 if (p[1] == 'i')
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1856 goto LscanReal; // 0Li
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1857 break; // 0L
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1858 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1859 if (p[1] == '.')
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1860 break; // 0..
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1861 // 0.
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1862 case 'i','f','F', // Imaginary and float literal suffixes.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1863 'e', 'E': // Float exponent.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1864 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1865 default:
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1866 if (*p == '_')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1867 goto LscanOctal; // 0_
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1868 else if (isdigit(*p))
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1869 {
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1870 if (p == '8' \|\| p == '9')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1871 goto Loctal_hasDecimalDigits; // 08 or 09
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1872 else
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1873 goto Loctal_enter_loop; // 0[0-7]
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1874 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1875 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1876
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1877 // Number 0
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1878 assert(p[-1] == '0');
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1879 assert(p != '_' && !isdigit(p));
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1880 assert(ulong_ == 0);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1881 isDecimal = true;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1882 goto Lfinalize;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1883
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1884 LscanInteger:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1885 assert(p != 0 && isdigit(p));
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1886 isDecimal = true;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1887 goto Lenter_loop_int;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1888 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1889 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1890 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1891 continue;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1892 if (!isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1893 break;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1894 Lenter_loop_int:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1895 if (ulong_ < ulong.max/10 \|\| (ulong_ == ulong.max/10 && *p <= '5'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1896 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1897 ulong_ *= 10;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1898 ulong_ += *p - '0';
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1899 continue;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1900 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1901 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1902 overflow = true;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1903 while (isdigit(*++p)) {}
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1904 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1905 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1906
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1907 // The number could be a float, so check overflow below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1908 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1909 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1910 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1911 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1912 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1913 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1914 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1915 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1916 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1917 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1918 goto LscanReal;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1919 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1920 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1921
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1922 if (overflow)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1923 error(t.start, MID.OverflowDecimalNumber);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1924
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1925 assert((isdigit(p[-1]) \|\| p[-1] == '_') && !isdigit(p) && p != '_');
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1926 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1927
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1928 LscanHex:
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1929 assert(digits == 0);
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	1930 assert(p == 'x' \|\| p == 'X');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1931 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1932 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1933 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1934 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1935 if (!ishexad(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1936 break;
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1937 ++digits;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1938 ulong_ *= 16;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1939 if (*p <= '9')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1940 ulong_ += *p - '0';
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1941 else if (*p <= 'F')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1942 ulong_ += *p - 'A' + 10;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1943 else
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1944 ulong_ += *p - 'a' + 10;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1945 }
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1946
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	1947 assert(ishexad(p[-1]) \|\| p[-1] == '_' \|\| p[-1] == 'x' \|\| p[-1] == 'X');
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1948 assert(!ishexad(p) && p != '_');
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1949
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1950 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1951 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1952 case '.':
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1953 if (p[1] == '.')
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1954 break;
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1955 case 'p', 'P':
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1956 return scanHexReal(t);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1957 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1958 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1959
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1960 if (digits == 0 \|\| digits > 16)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1961 error(t.start, digits == 0 ? MID.NoDigitsInHexNumber : MID.OverflowHexNumber);
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1962
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1963 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1964
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1965 LscanBinary:
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1966 assert(digits == 0);
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	1967 assert(p == 'b' \|\| p == 'B');
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1968 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1969 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1970 if (*++p == '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1971 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1972 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1973 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1974 }
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1975 else if (*p == '1')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1976 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1977 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1978 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1979 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1980 }
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1981 else if (*p == '_')
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1982 continue;
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1983 else
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1984 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1985 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1986
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1987 if (digits == 0 \|\| digits > 64)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1988 error(t.start, digits == 0 ? MID.NoDigitsInBinNumber : MID.OverflowBinaryNumber);
59 3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	1989
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	1990 assert(p[-1] == '0' \|\| p[-1] == '1' \|\| p[-1] == '_' \|\| p[-1] == 'b' \|\| p[-1] == 'B', p[-1] ~ "");
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1991 assert( !(p == '0' \|\| p == '1' \|\| *p == '_') );
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1992 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1993
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1994 LscanOctal:
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1995 assert(*p == '_');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1996 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1997 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1998 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1999 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2000 if (!isoctal(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2001 break;
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2002 Loctal_enter_loop:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2003 if (ulong_ < ulong.max/2 \|\| (ulong_ == ulong.max/2 && *p <= '1'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2004 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2005 ulong_ *= 8;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2006 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2007 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2008 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2009 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2010 overflow = true;
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2011 while (isoctal(*++p)) {}
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2012 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2013 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2014
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2015 bool hasDecimalDigits;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2016 if (isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2017 {
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2018 Loctal_hasDecimalDigits:
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2019 hasDecimalDigits = true;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2020 while (isdigit(*++p)) {}
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2021 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2022
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2023 // The number could be a float, so check errors below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2024 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2025 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2026 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2027 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2028 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2029 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2030 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2031 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2032 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2033 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2034 goto LscanReal;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2035 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2036 }
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2037
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2038 if (hasDecimalDigits)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2039 error(t.start, MID.OctalNumberHasDecimals);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2040
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2041 if (overflow)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2042 error(t.start, MID.OverflowOctalNumber);
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2043 // goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2044
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2045 Lfinalize:
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2046 enum Suffix
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2047 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2048 None = 0,
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2049 Unsigned = 1,
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2050 Long = 2
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2051 }
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2052
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2053 // Scan optional suffix: L, Lu, LU, u, uL, U or UL.
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2054 Suffix suffix;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2055 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2056 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2057 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2058 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2059 case 'L':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2060 if (suffix & Suffix.Long)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2061 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2062 suffix \|= Suffix.Long;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2063 ++p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2064 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2065 case 'u', 'U':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2066 if (suffix & Suffix.Unsigned)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2067 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2068 suffix \|= Suffix.Unsigned;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2069 ++p;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2070 continue;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2071 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2072 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2073 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2074 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2075 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2076
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2077 // Determine type of Integer.
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2078 switch (suffix)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2079 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2080 case Suffix.None:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2081 if (ulong_ & 0x8000_0000_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2082 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2083 if (isDecimal)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2084 error(t.start, MID.OverflowDecimalSign);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2085 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2086 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2087 else if (ulong_ & 0xFFFF_FFFF_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2088 t.type = TOK.Int64;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2089 else if (ulong_ & 0x8000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2090 t.type = isDecimal ? TOK.Int64 : TOK.Uint32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2091 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2092 t.type = TOK.Int32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2093 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2094 case Suffix.Unsigned:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2095 if (ulong_ & 0xFFFF_FFFF_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2096 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2097 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2098 t.type = TOK.Uint32;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2099 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2100 case Suffix.Long:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2101 if (ulong_ & 0x8000_0000_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2102 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2103 if (isDecimal)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2104 error(t.start, MID.OverflowDecimalSign);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2105 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2106 }
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2107 else
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2108 t.type = TOK.Int64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2109 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2110 case Suffix.Unsigned \| Suffix.Long:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2111 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2112 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2113 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2114 assert(0);
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2115 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2116 t.ulong_ = ulong_;
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	2117 t.end = p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2118 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2119 LscanReal:
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2120 scanReal(t);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2121 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2122 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2123
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2124 /*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2125 FloatLiteral:= Float[fFL]?i?
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2126 Float:= DecFloat \| HexFloat
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2127 DecFloat:= ([0-9][0-9_][.][0-9_]DecExponent?) \| [.][0-9][0-9_]DecExponent? \| [0-9][0-9_]DecExponent
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2128 DecExponent:= [eE][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2129 HexFloat:= 0[xX](HexDigits[.]HexDigits \| [.][0-9a-zA-Z]HexDigits? \| HexDigits)HexExponent
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2130 HexExponent:= [pP][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2131 */
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2132 void scanReal(ref Token t)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2133 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2134 if (*p == '.')
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2135 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2136 assert(p[1] != '.');
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2137 // This function was called by scan() or scanNumber().
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2138 while (isdigit(++p) \|\| p == '_') {}
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2139 }
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2140 else
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2141 // This function was called by scanNumber().
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2142 assert(delegate ()
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2143 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2144 switch (*p)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2145 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2146 case 'L':
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2147 if (p[1] != 'i')
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2148 return false;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2149 case 'i', 'f', 'F', 'e', 'E':
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2150 return true;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2151 default:
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2152 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2153 return false;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2154 }()
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2155 );
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2156
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2157 // Scan exponent.
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2158 if (p == 'e' \|\| p == 'E')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2159 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2160 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2161 if (p == '-' \|\| p == '+')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2162 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2163 if (isdigit(*p))
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2164 while (isdigit(++p) \|\| p == '_') {}
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2165 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2166 error(t.start, MID.FloatExpMustStartWithDigit);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2167 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2168
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2169 // Copy whole number and remove underscores from buffer.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2170 char[] buffer = t.start[0..p-t.start].dup;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2171 uint j;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2172 foreach (c; buffer)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2173 if (c != '_')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2174 buffer[j++] = c;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2175 buffer.length = j; // Adjust length.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2176 buffer ~= 0; // Terminate for C functions.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2177
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2178 finalizeFloat(t, buffer);
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2179 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2180
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2181 void scanHexReal(ref Token t)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2182 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2183 assert(p == '.' \|\| p == 'p' \|\| *p == 'P');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2184 MID mid;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2185 if (*p == '.')
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2186 while (ishexad(++p) \|\| p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2187 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2188 // Decimal exponent is required.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2189 if (p != 'p' && p != 'P')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2190 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2191 mid = MID.HexFloatExponentRequired;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2192 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2193 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2194 // Scan exponent
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2195 assert(p == 'p' \|\| p == 'P');
393 fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2196 ++p;
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2197 if (p == '+' \|\| p == '-')
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2198 ++p;
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2199 if (!isdigit(*p))
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2200 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2201 mid = MID.HexFloatExpMustStartWithDigit;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2202 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2203 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2204 while (isdigit(++p) \|\| p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2205 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2206 // Copy whole number and remove underscores from buffer.
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2207 char[] buffer = t.start[0..p-t.start].dup;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2208 uint j;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2209 foreach (c; buffer)
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2210 if (c != '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2211 buffer[j++] = c;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2212 buffer.length = j; // Adjust length.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2213 buffer ~= 0; // Terminate for C functions.
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2214 finalizeFloat(t, buffer);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2215 return;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2216 Lerr:
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2217 t.type = TOK.Float32;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2218 t.end = p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2219 error(t.start, mid);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2220 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2221
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2222 void finalizeFloat(ref Token t, string buffer)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2223 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2224 assert(buffer[$-1] == 0);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2225 // Float number is well-formed. Check suffixes and do conversion.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2226 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2227 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2228 case 'f', 'F':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2229 t.type = TOK.Float32;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2230 t.float_ = strtof(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2231 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2232 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2233 case 'L':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2234 t.type = TOK.Float80;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2235 t.real_ = strtold(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2236 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2237 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2238 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2239 t.type = TOK.Float64;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2240 t.double_ = strtod(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2241 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2242 if (*p == 'i')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2243 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2244 ++p;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2245 t.type += 3; // Switch to imaginary counterpart.
538 d0d40bcca9c6 Added assert() to Lexer.finalizeFloat(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 532 diff changeset	2246 assert(t.type == TOK.Imaginary32 \|\|
d0d40bcca9c6 Added assert() to Lexer.finalizeFloat(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 532 diff changeset	2247 t.type == TOK.Imaginary64 \|\|
d0d40bcca9c6 Added assert() to Lexer.finalizeFloat(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 532 diff changeset	2248 t.type == TOK.Imaginary80);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2249 }
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	2250 if (errno() == ERANGE)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2251 error(t.start, MID.OverflowFloatNumber);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2252 t.end = p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2253 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2254
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2255 /// Scan special token: #line Integer [Filespec] EndOfLine
360 b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence(). aziz parents: 350 diff changeset	2256 void scanSpecialTokenSequence(ref Token t)
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2257 {
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	2258 assert(*p == '#');
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2259 t.type = TOK.HashLine;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	2260 t.setWhitespaceFlag();
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2261
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2262 MID mid;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2263 auto errorAtColumn = p;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2264
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2265 ++p;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2266 if (p[0] != 'l' \|\| p[1] != 'i' \|\| p[2] != 'n' \|\| p[3] != 'e')
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2267 {
370 ae4afb66768f - Renamed findModule() to findModulePath(). aziz parents: 367 diff changeset	2268 mid = MID.ExpectedIdentifierSTLine;
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2269 goto Lerr;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2270 }
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2271 p += 3;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2272
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2273 // TODO: #line58"path/file" is legal. Require spaces?
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2274 // State.Space could be used for that purpose.
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2275 enum State
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2276 { /+Space,+/ Integer, Filespec, End }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2277
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2278 State state = State.Integer;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2279
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2280 while (!isEndOfLine(++p))
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2281 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2282 if (isspace(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2283 continue;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2284 if (state == State.Integer)
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2285 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2286 if (!isdigit(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2287 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2288 errorAtColumn = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2289 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2290 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2291 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2292 t.tokLineNum = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2293 scan(*t.tokLineNum);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2294 if (t.tokLineNum.type != TOK.Int32 && t.tokLineNum.type != TOK.Uint32)
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2295 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2296 errorAtColumn = t.tokLineNum.start;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2297 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2298 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2299 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2300 --p; // Go one back because scan() advanced p past the integer.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2301 state = State.Filespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2302 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2303 else if (state == State.Filespec)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2304 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2305 if (*p != '"')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2306 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2307 errorAtColumn = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2308 mid = MID.ExpectedFilespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2309 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2310 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2311 t.tokLineFilespec = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2312 t.tokLineFilespec.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2313 t.tokLineFilespec.type = TOK.Filespec;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	2314 t.tokLineFilespec.setWhitespaceFlag();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2315 while (*++p != '"')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2316 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2317 if (isEndOfLine(p))
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2318 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2319 errorAtColumn = t.tokLineFilespec.start;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2320 mid = MID.UnterminatedFilespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2321 t.tokLineFilespec.end = p;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2322 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2323 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2324 isascii(*p) \|\| decodeUTF8();
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2325 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2326 auto start = t.tokLineFilespec.start +1; // +1 skips '"'
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2327 t.tokLineFilespec.str = start[0 .. p - start];
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2328 t.tokLineFilespec.end = p + 1;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2329 state = State.End;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2330 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2331 else/+ if (state == State.End)+/
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2332 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2333 mid = MID.UnterminatedSpecialToken;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2334 goto Lerr;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2335 }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2336 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2337 assert(isEndOfLine(p));
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2338
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2339 if (state == State.Integer)
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2340 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2341 errorAtColumn = p;
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2342 mid = MID.ExpectedIntegerAfterSTLine;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2343 goto Lerr;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2344 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2345
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2346 // Evaluate #line only when not in token string.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2347 if (!inTokenString && t.tokLineNum)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2348 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2349 this.lineNum_hline = this.lineNum - t.tokLineNum.uint_ + 1;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2350 if (t.tokLineFilespec)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2351 this.errorPath = t.tokLineFilespec.str;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2352 }
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2353 t.end = p;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2354
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2355 return;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2356 Lerr:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2357 t.end = p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2358 error(errorAtColumn, mid);
371 01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	2359 }
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	2360
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2361 /++
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2362 Insert an empty dummy token before t.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2363 Useful in the parsing phase for representing a node in the AST
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2364 that doesn't consume an actual token from the source text.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2365 +/
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2366 Token* insertEmptyTokenBefore(Token* t)
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2367 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2368 assert(t !is null && t.prev !is null);
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2369 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type));
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2370 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type));
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2371
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2372 auto prev_t = t.prev;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2373 auto new_t = new Token;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2374 new_t.type = TOK.Empty;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2375 new_t.start = new_t.end = prev_t.end;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2376 // Link in new token.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2377 prev_t.next = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2378 new_t.prev = prev_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2379 new_t.next = t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2380 t.prev = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2381 return new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2382 }
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	2383
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2384 uint errorLineNumber(uint lineNum)
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	2385 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2386 return lineNum - this.lineNum_hline;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	2387 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	2388
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2389 void error(char* columnPos, MID mid, ...)
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2390 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2391 error_(this.lineNum, this.lineBegin, columnPos, mid, _arguments, _argptr);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2392 }
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2393
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2394 void error(uint lineNum, char* lineBegin, char* columnPos, MID mid, ...)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2395 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2396 error_(lineNum, lineBegin, columnPos, mid, _arguments, _argptr);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2397 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2398
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2399 void error_(uint lineNum, char* lineBegin, char* columnPos, MID mid,
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2400 TypeInfo[] _arguments, void* _argptr)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2401 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2402 lineNum = this.errorLineNumber(lineNum);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2403 auto location = new Location(errorPath, lineNum, lineBegin, columnPos);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2404 auto msg = Format(_arguments, _argptr, GetMsg(mid));
520 f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	2405 auto error = new LexerError(location, msg);
f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	2406 errors ~= error;
f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	2407 if (infoMan !is null)
f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	2408 infoMan ~= error;
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2409 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2410
306 9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2411 Token* getTokens()
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	2412 {
306 9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2413 while (nextToken() != TOK.EOF)
9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2414 {}
9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2415 return head;
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	2416 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2417
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2418 /// Scan the whole text until EOF is encountered.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2419 void scanAll()
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2420 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2421 while (nextToken() != TOK.EOF)
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2422 {}
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2423 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2424
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2425 /// HEAD -> Newline -> First Token
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2426 Token* firstToken()
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2427 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2428 return this.head.next.next;
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2429 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2430
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2431 static void loadKeywords(ref Identifier[string] table)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2432 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2433 foreach(k; keywords)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2434 table[k.str] = k;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2435 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2436
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2437 /// Returns true if str is a valid D identifier.
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2438 static bool isIdentifierString(char[] str)
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2439 {
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2440 if (str.length == 0 \|\| isdigit(str[0]))
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2441 return false;
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2442 size_t idx;
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2443 do
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2444 {
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2445 auto c = dil.Unicode.decode(str, idx);
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2446 if (c == ERROR_CHAR \|\| !(isident(c) \|\| !isascii(c) && isUniAlpha(c)))
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2447 return false;
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2448 } while (idx < str.length)
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2449 return true;
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2450 }
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2451
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2452 /// Returns true if str is a keyword or a special token (__FILE__, __LINE__ etc.)
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2453 static bool isReservedIdentifier(char[] str)
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2454 {
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2455 if (str.length == 0)
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2456 return false;
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2457
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2458 static Identifier[string] reserved_ids_table;
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2459 if (reserved_ids_table is null)
499 52447db67938 Implemented global table of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 498 diff changeset	2460 Lexer.loadKeywords(reserved_ids_table);
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2461
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2462 if (!isIdentifierString(str))
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2463 return false;
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2464
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2465 return (str in reserved_ids_table) !is null;
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2466 }
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2467
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2468 /++
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2469 Returns true if the current character to be decoded is
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2470 a Unicode alpha character.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2471 The current pointer 'p' is not advanced if false is returned.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2472 +/
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2473 bool isUnicodeAlpha()
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2474 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2475 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2476 char* p = this.p;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2477 dchar d = *p;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2478 ++p; // Move to second byte.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2479 // Error if second byte is not a trail byte.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2480 if (!isTrailByte(*p))
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2481 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2482 // Check for overlong sequences.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2483 switch (d)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2484 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2485 case 0xE0, 0xF0, 0xF8, 0xFC:
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2486 if ((*p & d) == 0x80)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2487 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2488 default:
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2489 if ((d & 0xFE) == 0xC0) // 1100000x
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2490 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2491 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2492 const char[] checkNextByte = "if (!isTrailByte(*++p))"
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2493 " return false;";
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2494 const char[] appendSixBits = "d = (d << 6) \| *p & 0b0011_1111;";
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2495 // Decode
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2496 if ((d & 0b1110_0000) == 0b1100_0000)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2497 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2498 d &= 0b0001_1111;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2499 mixin(appendSixBits);
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2500 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2501 else if ((d & 0b1111_0000) == 0b1110_0000)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2502 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2503 d &= 0b0000_1111;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2504 mixin(appendSixBits ~
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2505 checkNextByte ~ appendSixBits);
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2506 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2507 else if ((d & 0b1111_1000) == 0b1111_0000)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2508 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2509 d &= 0b0000_0111;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2510 mixin(appendSixBits ~
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2511 checkNextByte ~ appendSixBits ~
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2512 checkNextByte ~ appendSixBits);
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2513 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2514 else
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2515 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2516
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2517 assert(isTrailByte(*p));
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2518 if (!isValidChar(d) \|\| !isUniAlpha(d))
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2519 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2520 // Only advance pointer if this is a Unicode alpha character.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2521 this.p = p;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2522 return true;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2523 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2524
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2525 /// Decodes the next UTF-8 sequence.
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2526 dchar decodeUTF8()
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2527 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2528 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2529 char* p = this.p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2530 dchar d = *p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2531
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2532 ++p; // Move to second byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2533 // Error if second byte is not a trail byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2534 if (!isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2535 goto Lerr2;
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2536
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2537 // Check for overlong sequences.
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2538 switch (d)
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2539 {
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2540 case 0xE0, // 11100000 100xxxxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2541 0xF0, // 11110000 1000xxxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2542 0xF8, // 11111000 10000xxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2543 0xFC: // 11111100 100000xx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2544 if ((*p & d) == 0x80)
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2545 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2546 default:
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2547 if ((d & 0xFE) == 0xC0) // 1100000x
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2548 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2549 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2550
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2551 const char[] checkNextByte = "if (!isTrailByte(*++p))"
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2552 " goto Lerr2;";
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2553 const char[] appendSixBits = "d = (d << 6) \| *p & 0b0011_1111;";
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2554
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2555 // Decode
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2556 if ((d & 0b1110_0000) == 0b1100_0000)
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2557 { // 110xxxxx 10xxxxxx
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2558 d &= 0b0001_1111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2559 mixin(appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2560 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2561 else if ((d & 0b1111_0000) == 0b1110_0000)
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2562 { // 1110xxxx 10xxxxxx 10xxxxxx
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2563 d &= 0b0000_1111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2564 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2565 checkNextByte ~ appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2566 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2567 else if ((d & 0b1111_1000) == 0b1111_0000)
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2568 { // 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2569 d &= 0b0000_0111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2570 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2571 checkNextByte ~ appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2572 checkNextByte ~ appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2573 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2574 else
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2575 // 5 and 6 byte UTF-8 sequences are not allowed yet.
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2576 // 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2577 // 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2578 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2579
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2580 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2581
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2582 if (!isValidChar(d))
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2583 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2584 Lerr:
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2585 // Three cases:
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2586 // *) the UTF-8 sequence was successfully decoded but the resulting
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2587 // character is invalid.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2588 // p points to last trail byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2589 // *) the UTF-8 sequence is overlong.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2590 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2591 // *) the UTF-8 sequence has more than 4 bytes or starts with
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2592 // a trail byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2593 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2594 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2595 // Move to next ASCII character or lead byte of a UTF-8 sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2596 while (p < (end-1) && isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2597 ++p;
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2598 --p;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2599 assert(!isTrailByte(p[1]));
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2600 Lerr2:
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2601 d = REPLACEMENT_CHAR;
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2602 error(this.p, MID.InvalidUTF8Sequence);
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2603 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2604
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2605 this.p = p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2606 return d;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2607 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2608
519 9ebc799c7dc5 Fixes in dil.Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 514 diff changeset	2609 static void encodeUTF8(ref char[] str, dchar d)
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2610 {
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2611 assert(!isascii(d), "check for ASCII char before calling encodeUTF8().");
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2612 assert(isValidChar(d), "check if character is valid before calling encodeUTF8().");
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2613
519 9ebc799c7dc5 Fixes in dil.Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 514 diff changeset	2614 char[6] b = void;
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2615 if (d < 0x800)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2616 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2617 b[0] = 0xC0 \| (d >> 6);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2618 b[1] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2619 str ~= b[0..2];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2620 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2621 else if (d < 0x10000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2622 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2623 b[0] = 0xE0 \| (d >> 12);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2624 b[1] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2625 b[2] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2626 str ~= b[0..3];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2627 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2628 else if (d < 0x200000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2629 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2630 b[0] = 0xF0 \| (d >> 18);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2631 b[1] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2632 b[2] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2633 b[3] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2634 str ~= b[0..4];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2635 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2636 /+ // There are no 5 and 6 byte UTF-8 sequences yet.
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2637 else if (d < 0x4000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2638 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2639 b[0] = 0xF8 \| (d >> 24);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2640 b[1] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2641 b[2] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2642 b[3] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2643 b[4] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2644 str ~= b[0..5];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2645 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2646 else if (d < 0x80000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2647 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2648 b[0] = 0xFC \| (d >> 30);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2649 b[1] = 0x80 \| ((d >> 24) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2650 b[2] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2651 b[3] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2652 b[4] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2653 b[5] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2654 str ~= b[0..6];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2655 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2656 +/
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2657 else
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2658 assert(0);
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2659 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	2660 }
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2661
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2662 unittest
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2663 {
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	2664 Stdout("Testing Lexer.\n");
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2665 struct Pair
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2666 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2667 char[] tokenText;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2668 TOK type;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2669 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2670 static Pair[] pairs = [
494 9a7ca8c56e59 Refactored a few things in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	2671 {"#!äöüß", TOK.Shebang}, {"\n", TOK.Newline},
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2672 {"//çay", TOK.Comment}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2673 {"&", TOK.AndBinary},
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2674 {"/çağ/", TOK.Comment}, {"&&", TOK.AndLogical},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2675 {"/+çak+/", TOK.Comment}, {"&=", TOK.AndAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2676 {">", TOK.Greater}, {"+", TOK.Plus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2677 {">=", TOK.GreaterEqual}, {"++", TOK.PlusPlus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2678 {">>", TOK.RShift}, {"+=", TOK.PlusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2679 {">>=", TOK.RShiftAssign}, {"-", TOK.Minus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2680 {">>>", TOK.URShift}, {"--", TOK.MinusMinus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2681 {">>>=", TOK.URShiftAssign}, {"-=", TOK.MinusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2682 {"<", TOK.Less}, {"=", TOK.Assign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2683 {"<=", TOK.LessEqual}, {"==", TOK.Equal},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2684 {"<>", TOK.LorG}, {"~", TOK.Tilde},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2685 {"<>=", TOK.LorEorG}, {"~=", TOK.CatAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2686 {"<<", TOK.LShift}, {"*", TOK.Mul},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2687 {"<<=", TOK.LShiftAssign}, {"*=", TOK.MulAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2688 {"!", TOK.Not}, {"/", TOK.Div},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2689 {"!=", TOK.NotEqual}, {"/=", TOK.DivAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2690 {"!<", TOK.UorGorE}, {"^", TOK.Xor},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2691 {"!>", TOK.UorLorE}, {"^=", TOK.XorAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2692 {"!<=", TOK.UorG}, {"%", TOK.Mod},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2693 {"!>=", TOK.UorL}, {"%=", TOK.ModAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2694 {"!<>", TOK.UorE}, {"(", TOK.LParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2695 {"!<>=", TOK.Unordered}, {")", TOK.RParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2696 {".", TOK.Dot}, {"[", TOK.LBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2697 {"..", TOK.Slice}, {"]", TOK.RBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2698 {"...", TOK.Ellipses}, {"{", TOK.LBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2699 {"\|", TOK.OrBinary}, {"}", TOK.RBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2700 {"\|\|", TOK.OrLogical}, {":", TOK.Colon},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2701 {"\|=", TOK.OrAssign}, {";", TOK.Semicolon},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2702 {"?", TOK.Question}, {",", TOK.Comma},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2703 {"$", TOK.Dollar}, {"cam", TOK.Identifier},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2704 {"çay", TOK.Identifier}, {".0", TOK.Float64},
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2705 {"0", TOK.Int32}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2706 {"\r", TOK.Newline}, {"\r\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2707 {"\u2028", TOK.Newline}, {"\u2029", TOK.Newline}
40 9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2708 ];
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2709
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2710 char[] src;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2711
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2712 // Join all token texts into a single string.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2713 foreach (i, pair; pairs)
494 9a7ca8c56e59 Refactored a few things in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	2714 if (pair.type == TOK.Comment && pair.tokenText[1] == '/' \|\| // Line comment.
9a7ca8c56e59 Refactored a few things in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	2715 pair.type == TOK.Shebang)
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2716 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2717 assert(pairs[i+1].type == TOK.Newline); // Must be followed by a newline.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2718 src ~= pair.tokenText;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2719 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2720 else
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2721 src ~= pair.tokenText ~ " ";
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2722
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2723 auto lx = new Lexer(src, "");
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2724 auto token = lx.getTokens();
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2725
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2726 uint i;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2727 assert(token == lx.head);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2728 assert(token.next.type == TOK.Newline);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2729 token = token.next.next;
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2730 do
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2731 {
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2732 assert(i < pairs.length);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2733 assert(token.srcText == pairs[i].tokenText, Format("Scanned '{0}' but expected '{1}'", token.srcText, pairs[i].tokenText));
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2734 ++i;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2735 token = token.next;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2736 } while (token.type != TOK.EOF)
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	2737 }
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2738
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2739 unittest
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2740 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2741 Stdout("Testing method Lexer.peek()\n");
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2742 string sourceText = "unittest { }";
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2743 auto lx = new Lexer(sourceText, null);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2744
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2745 auto next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2746 lx.peek(next);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2747 assert(next.type == TOK.Newline);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2748 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2749 assert(next.type == TOK.Unittest);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2750 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2751 assert(next.type == TOK.LBrace);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2752 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2753 assert(next.type == TOK.RBrace);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2754 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2755 assert(next.type == TOK.EOF);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2756
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2757 lx = new Lexer("", null);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2758 next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2759 lx.peek(next);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2760 assert(next.type == TOK.Newline);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2761 lx.peek(next);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2762 assert(next.type == TOK.EOF);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2763 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2764
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2765 unittest
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2766 {
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2767 // Numbers unittest
68 7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2768 // 0L 0ULi 0_L 0_UL 0x0U 0x0p2 0_Fi 0_e2 0_F 0_i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2769 // 0u 0U 0uL 0UL 0L 0LU 0Lu
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2770 // 0Li 0f 0F 0fi 0Fi 0i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2771 // 0b_1_LU 0b1000u
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2772 // 0x232Lu
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2773 }
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2774
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2775 /// ASCII character properties table.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2776 static const int ptable[256] = [
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2777 0, 0, 0, 0, 0, 0, 0, 0, 0,32, 0,32,32, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2778 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2779 32, 0, 0x2200, 0, 0, 0, 0, 0x2700, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2780 7, 7, 7, 7, 7, 7, 7, 7, 6, 6, 0, 0, 0, 0, 0, 0x3f00,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2781 0,12,12,12,12,12,12, 8, 8, 8, 8, 8, 8, 8, 8, 8,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2782 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0, 0x5c00, 0, 0,16,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2783 0, 0x70c, 0x80c,12,12,12, 0xc0c, 8, 8, 8, 8, 8, 8, 8, 0xa08, 8,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2784 8, 8, 0xd08, 8, 0x908, 8, 0xb08, 8, 8, 8, 8, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2785 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2786 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2787 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2788 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2789 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2790 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2791 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2792 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2793 ];
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2794
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2795 enum CProperty
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2796 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2797 Octal = 1,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2798 Digit = 1<<1,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2799 Hex = 1<<2,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2800 Alpha = 1<<3,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2801 Underscore = 1<<4,
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2802 Whitespace = 1<<5
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2803 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2804
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2805 const uint EVMask = 0xFF00; // Bit mask for escape value
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2806
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2807 private alias CProperty CP;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2808 int isoctal(char c) { return ptable[c] & CP.Octal; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2809 int isdigit(char c) { return ptable[c] & CP.Digit; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2810 int ishexad(char c) { return ptable[c] & CP.Hex; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2811 int isalpha(char c) { return ptable[c] & CP.Alpha; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2812 int isalnum(char c) { return ptable[c] & (CP.Alpha \| CP.Digit); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2813 int isidbeg(char c) { return ptable[c] & (CP.Alpha \| CP.Underscore); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2814 int isident(char c) { return ptable[c] & (CP.Alpha \| CP.Underscore \| CP.Digit); }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2815 int isspace(char c) { return ptable[c] & CP.Whitespace; }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2816 int char2ev(char c) { return ptable[c] >> 8; /(ptable[c] & EVMask) >> 8;/ }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2817 int isascii(uint c) { return c < 128; }
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2818
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2819 version(gen_ptable)
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2820 static this()
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2821 {
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2822 alias ptable p;
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2823 assert(p.length == 256);
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2824 // Initialize character properties table.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2825 for (int i; i < p.length; ++i)
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2826 {
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2827 p[i] = 0; // Reset
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2828 if ('0' <= i && i <= '7')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2829 p[i] \|= CP.Octal;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2830 if ('0' <= i && i <= '9')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2831 p[i] \|= CP.Digit;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2832 if (isdigit(i) \|\| 'a' <= i && i <= 'f' \|\| 'A' <= i && i <= 'F')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2833 p[i] \|= CP.Hex;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2834 if ('a' <= i && i <= 'z' \|\| 'A' <= i && i <= 'Z')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2835 p[i] \|= CP.Alpha;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2836 if (i == '_')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2837 p[i] \|= CP.Underscore;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2838 if (i == ' ' \|\| i == '\t' \|\| i == '\v' \|\| i == '\f')
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2839 p[i] \|= CP.Whitespace;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2840 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2841 // Store escape sequence values in second byte.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2842 assert(CProperty.max <= ubyte.max, "character property flags and escape value byte overlap.");
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2843 p['\''] \|= 39 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2844 p['"'] \|= 34 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2845 p['?'] \|= 63 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2846 p['\\'] \|= 92 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2847 p['a'] \|= 7 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2848 p['b'] \|= 8 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2849 p['f'] \|= 12 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2850 p['n'] \|= 10 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2851 p['r'] \|= 13 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2852 p['t'] \|= 9 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2853 p['v'] \|= 11 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2854 // Print a formatted array literal.
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2855 char[] array = "[\n";
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2856 foreach (i, c; ptable)
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2857 {
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2858 array ~= Format((c>255?" 0x{0:x},":"{0,2},"), c) ~ (((i+1) % 16) ? "":"\n");
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2859 }
5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2860 array[$-2..$] = "\n]";
402 22d65b2bef4f Migrated the code that generates the ptable to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 397 diff changeset	2861 Stdout(array).newline;
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2862 }

Mercurial > projects > dil

annotate trunk/src/dil/lexer/Lexer.d @ 601:9f96fd51cb41