projects/dil: trunk/src/dil/lexer/Lexer.d annotate

annotate trunk/src/dil/lexer/Lexer.d @ 789:c1d5cfd7aa44

Implemented string literal conversion. Removed two MID messages. Added MSG.InvalidUTF8SequenceInString. Added toUTF16() and toUTF32(). Fixed escape sequences. Added formatBytes() and findInvalidUTF8Sequence().

author	Aziz K?ksal <aziz.koeksal@gmail.com>
date	Mon, 25 Feb 2008 02:56:22 +0100
parents	580d4ca9f1ff
children	cf2ad5df025c

rev	line source
0 8ba2570de175 Initial import. aziz parents: diff changeset	1 /++
8ba2570de175 Initial import. aziz parents: diff changeset	2 Author: Aziz Köksal
249 32d354584b28 - Upgraded license notices to GPL3. aziz parents: 239 diff changeset	3 License: GPL3
0 8ba2570de175 Initial import. aziz parents: diff changeset	4 +/
576 0df647660e76 Moved Lexer.d to new package 'lexer'. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 552 diff changeset	5 module dil.lexer.Lexer;
577 9e811db780a6 Moved LexerFuncs.d to package 'lexer'. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 576 diff changeset	6
596 39fac5531b85 Moved dil.Token to dil.lexer.Token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 577 diff changeset	7 import dil.lexer.Token;
599 c4cdea3a65ba Moved dil.Keywords to dil.lexer.Keywords. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 596 diff changeset	8 import dil.lexer.Keywords;
600 041eae272362 Moved dil.Identifier to dil.lexer.Identifier. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 599 diff changeset	9 import dil.lexer.Identifier;
601 9f96fd51cb41 Moved dil.IdTable to dil.lexer.IdTable. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 600 diff changeset	10 import dil.lexer.IdTable;
327 a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	11 import dil.Information;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	12 import dil.Messages;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	13 import dil.HtmlEntities;
464 325714d8aa6c Added new module with compiler constants to resolve import cycle. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 434 diff changeset	14 import dil.CompilerInfo;
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	15 import dil.Unicode;
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	16 import dil.SourceText;
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	17 import dil.Time;
737 f88b5285b86b Implemented DDocEmitter. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 715 diff changeset	18 import common;
f88b5285b86b Implemented DDocEmitter. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 715 diff changeset	19
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	20 import tango.stdc.stdlib : strtof, strtod, strtold;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	21 import tango.stdc.errno : errno, ERANGE;
0 8ba2570de175 Initial import. aziz parents: diff changeset	22
577 9e811db780a6 Moved LexerFuncs.d to package 'lexer'. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 576 diff changeset	23 public import dil.lexer.Funcs;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	24
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	25 /// The Lexer analyzes the characters of a source text and
580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	26 /// produces a doubly-linked list of tokens.
0 8ba2570de175 Initial import. aziz parents: diff changeset	27 class Lexer
8ba2570de175 Initial import. aziz parents: diff changeset	28 {
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	29 SourceText srcText; /// The source text.
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	30 char* p; /// Points to the current character in the source text.
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	31 char* end; /// Points one character past the end of the source text.
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	32
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	33 Token* head; /// The head of the doubly linked token list.
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	34 Token* tail; /// The tail of the linked list. Set in scan().
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	35 Token* token; /// Points to the current token in the token list.
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	36
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	37 // Members used for error messages:
532 50e64bab9c7a Renamed InformationManager to InfoManager. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 520 diff changeset	38 InfoManager infoMan;
514 6ddff941862a Added new error classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 510 diff changeset	39 LexerError[] errors;
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	40 /// Always points to the first character of the current line.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	41 char* lineBegin;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	42 // Token* newline; /// Current newline token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	43 uint lineNum = 1; /// Current, actual source text line number.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	44 uint lineNum_hline; /// Line number set by #line.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	45 uint inTokenString; /// > 0 if inside q{ }
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	46 NewlineData.FilePaths* filePaths;
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	47
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	48 /// Construct a Lexer object.
580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	49 /// Params:
580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	50 /// srcText = the UTF-8 source code.
580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	51 /// infoMan = used for collecting error messages.
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	52 this(SourceText srcText, InfoManager infoMan = null)
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	53 {
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	54 this.srcText = srcText;
520 f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	55 this.infoMan = infoMan;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	56
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	57 assert(text.length && text[$-1] == 0, "source text has no sentinel character");
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	58 this.p = text.ptr;
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	59 this.end = this.p + text.length;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	60 this.lineBegin = this.p;
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	61
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	62 this.head = new Token;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	63 this.head.kind = TOK.HEAD;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	64 this.head.start = this.head.end = this.p;
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	65 this.token = this.head;
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	66 // Initialize this.filePaths.
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	67 newFilePath(this.srcText.filePath);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	68 // Add a newline as the first token after the head.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	69 auto newline = new Token;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	70 newline.kind = TOK.Newline;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	71 newline.setWhitespaceFlag();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	72 newline.start = newline.end = this.p;
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	73 newline.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	74 newline.newline.oriLineNum = 1;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	75 newline.newline.setLineNum = 0;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	76 // Link in.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	77 this.token.next = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	78 newline.prev = this.token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	79 this.token = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	80 // this.newline = newline;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	81 scanShebang();
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	82 }
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	83
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	84 ~this()
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	85 {
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	86 auto token = head.next;
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	87 while (token !is null)
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	88 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	89 assert(token.kind == TOK.EOF ? token == tail && token.next is null : 1);
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	90 delete token.prev;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	91 token = token.next;
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	92 }
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	93 delete tail;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	94 }
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	95
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	96 char[] text()
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	97 {
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	98 return srcText.data;
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	99 }
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	100
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	101 /// The "shebang" may optionally appear once at the beginning of a file.
580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	102 /// Regexp: #![^\EndOfLine]*
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	103 void scanShebang()
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	104 {
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	105 if (*p == '#' && p[1] == '!')
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	106 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	107 auto t = new Token;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	108 t.kind = TOK.Shebang;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	109 t.setWhitespaceFlag();
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	110 t.start = p;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	111 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	112 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	113 isascii(*p) \|\| decodeUTF8();
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	114 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	115 this.token.next = t;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	116 t.prev = this.token;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	117 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	118 }
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	119
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	120 void finalizeSpecialToken(ref Token t)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	121 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	122 assert(t.srcText[0..2] == "__");
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	123 switch (t.kind)
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	124 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	125 case TOK.FILE:
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	126 t.str = this.filePaths.setPath;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	127 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	128 case TOK.LINE:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	129 t.uint_ = this.errorLineNumber(this.lineNum);
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	130 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	131 case TOK.DATE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	132 TOK.TIME,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	133 TOK.TIMESTAMP:
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	134 auto time_str = Time.toString();
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	135 switch (t.kind)
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	136 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	137 case TOK.DATE:
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	138 time_str = Time.month_day(time_str) ~ ' ' ~ Time.year(time_str); break;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	139 case TOK.TIME:
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	140 time_str = Time.time(time_str); break;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	141 case TOK.TIMESTAMP:
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	142 break; // time_str is the timestamp.
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	143 default: assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	144 }
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	145 time_str ~= '\0'; // Terminate with a zero.
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	146 t.str = time_str;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	147 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	148 case TOK.VENDOR:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	149 t.str = VENDOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	150 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	151 case TOK.VERSION:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	152 t.uint_ = VERSION_MAJOR*1000 + VERSION_MINOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	153 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	154 default:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	155 assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	156 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	157 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	158
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	159 /// Sets a new file path.
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	160 void newFilePath(char[] newPath)
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	161 {
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	162 auto paths = new NewlineData.FilePaths;
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	163 paths.oriPath = this.srcText.filePath;
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	164 paths.setPath = newPath;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	165 this.filePaths = paths;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	166 }
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	167
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	168 private void setLineBegin(char* p)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	169 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	170 // Check that we can look behind one character.
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	171 assert((p-1) >= text.ptr && p < end);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	172 // Check that previous character is a newline.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	173 assert(isNewlineEnd(p - 1));
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	174 this.lineBegin = p;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	175 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	176
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	177 private void scanNext(ref Token* t)
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	178 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	179 assert(t !is null);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	180 if (t.next)
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	181 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	182 t = t.next;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	183 // if (t.kind == TOK.Newline)
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	184 // this.newline = t;
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	185 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	186 else if (t != this.tail)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	187 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	188 Token* new_t = new Token;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	189 scan(*new_t);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	190 new_t.prev = t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	191 t.next = new_t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	192 t = new_t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	193 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	194 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	195
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	196 /// Advance t one token forward.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	197 void peek(ref Token* t)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	198 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	199 scanNext(t);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	200 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	201
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	202 /// Advance to the next token in the source text.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	203 TOK nextToken()
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	204 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	205 scanNext(this.token);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	206 return this.token.kind;
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	207 }
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	208
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	209 /// Returns true if p points to the last character of a Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	210 bool isNewlineEnd(char* p)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	211 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	212 if (p == '\n' \|\| p == '\r')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	213 return true;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	214 if (p == LS[2] \|\| p == PS[2])
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	215 if ((p-2) >= text.ptr)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	216 if (p[-1] == LS[1] && p[-2] == LS[0])
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	217 return true;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	218 return false;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	219 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	220
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	221 /++
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	222 This is the old scan method.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	223 TODO: profile old and new to see which one is faster.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	224 +/
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	225 public void scan(ref Token t)
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	226 in
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	227 {
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	228 assert(text.ptr <= p && p < end);
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	229 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	230 out
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	231 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	232 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	233 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	234 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	235 body
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	236 {
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	237 // Scan whitespace.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	238 if (isspace(*p))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	239 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	240 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	241 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	242 {}
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	243 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	244
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	245 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	246 uint c = *p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	247 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	248 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	249 // Newline.
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	250 switch (*p)
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	251 {
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	252 case '\r':
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	253 if (p[1] == '\n')
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	254 ++p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	255 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	256 assert(isNewlineEnd(p));
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	257 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	258 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	259 setLineBegin(p);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	260 // this.newline = &t;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	261 t.kind = TOK.Newline;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	262 t.setWhitespaceFlag();
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	263 t.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	264 t.newline.oriLineNum = lineNum;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	265 t.newline.setLineNum = lineNum_hline;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	266 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	267 return;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	268 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	269 if (isUnicodeNewline(p))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	270 {
412 fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	271 ++p; ++p;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	272 goto case '\n';
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	273 }
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	274 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	275 // Identifier or string literal.
13 e5211758b63c - Added isidbeg() function. aziz parents: 12 diff changeset	276 if (isidbeg(c))
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	277 {
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	278 if (c == 'r' && p[1] == '"' && ++p)
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	279 return scanRawStringLiteral(t);
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	280 if (c == 'x' && p[1] == '"')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	281 return scanHexStringLiteral(t);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	282 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	283 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	284 if (c == 'q' && p[1] == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	285 return scanDelimitedStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	286 if (c == 'q' && p[1] == '{')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	287 return scanTokenStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	288 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	289 // Scan identifier.
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	290 Lidentifier:
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	291 do
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	292 { c = *++p; }
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	293 while (isident(c) \|\| !isascii(c) && isUnicodeAlpha())
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	294
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	295 t.end = p;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	296
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 499 diff changeset	297 auto id = IdTable.lookup(t.srcText);
769 5e3ef1b2011c Added and improved documentation. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 764 diff changeset	298 t.kind = id.kind;
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 499 diff changeset	299 t.ident = id;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	300
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	301 if (t.kind == TOK.Identifier \|\| t.isKeyword)
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	302 return;
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	303 else if (t.isSpecialToken)
0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	304 finalizeSpecialToken(t);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	305 else if (t.kind == TOK.EOF)
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	306 {
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	307 tail = &t;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	308 assert(t.srcText == "__EOF__");
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	309 }
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	310 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	311 assert(0, "unexpected token type: " ~ Token.toString(t.kind));
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	312 return;
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	313 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	314
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	315 if (isdigit(c))
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	316 return scanNumber(t);
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	317
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	318 if (c == '/')
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	319 {
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	320 c = *++p;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	321 switch(c)
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	322 {
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	323 case '=':
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	324 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	325 t.kind = TOK.DivAssign;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	326 t.end = p;
cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	327 return;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	328 case '+':
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	329 return scanNestedComment(t);
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	330 case '*':
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	331 return scanBlockComment(t);
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	332 case '/':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	333 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	334 isascii(*p) \|\| decodeUTF8();
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	335 t.kind = TOK.Comment;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	336 t.setWhitespaceFlag();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	337 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	338 return;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	339 default:
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	340 t.kind = TOK.Div;
10 3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	341 t.end = p;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	342 return;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	343 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	344 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	345
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	346 switch (c)
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	347 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	348 case '\'':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	349 return scanCharacterLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	350 case '`':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	351 return scanRawStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	352 case '"':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	353 return scanNormalStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	354 case '\\':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	355 char[] buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	356 do
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	357 {
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	358 bool isBinary;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	359 c = scanEscapeSequence(isBinary);
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	360 if (isascii(c) \|\| isBinary)
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	361 buffer ~= c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	362 else
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	363 encodeUTF8(buffer, c);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	364 } while (*p == '\\')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	365 buffer ~= 0;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	366 t.kind = TOK.String;
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	367 t.str = buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	368 t.end = p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	369 return;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	370 case '>': /* > >= >> >>= >>> >>>= */
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	371 c = *++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	372 switch (c)
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	373 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	374 case '=':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	375 t.kind = TOK.GreaterEqual;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	376 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	377 case '>':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	378 if (p[1] == '>')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	379 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	380 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	381 if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	382 { ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	383 t.kind = TOK.URShiftAssign;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	384 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	385 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	386 t.kind = TOK.URShift;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	387 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	388 else if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	389 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	390 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	391 t.kind = TOK.RShiftAssign;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	392 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	393 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	394 t.kind = TOK.RShift;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	395 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	396 default:
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	397 t.kind = TOK.Greater;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	398 goto Lcommon2;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	399 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	400 assert(0);
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	401 case '<': /* < <= <> <>= << <<= */
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	402 c = *++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	403 switch (c)
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	404 {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	405 case '=':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	406 t.kind = TOK.LessEqual;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	407 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	408 case '<':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	409 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	410 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	411 t.kind = TOK.LShiftAssign;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	412 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	413 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	414 t.kind = TOK.LShift;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	415 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	416 case '>':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	417 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	418 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	419 t.kind = TOK.LorEorG;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	420 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	421 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	422 t.kind = TOK.LorG;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	423 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	424 default:
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	425 t.kind = TOK.Less;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	426 goto Lcommon2;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	427 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	428 assert(0);
37 7f3bcb97d017 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 36 diff changeset	429 case '!': /* ! !< !> !<= !>= !<> !<>= */
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	430 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	431 switch (c)
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	432 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	433 case '<':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	434 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	435 if (c == '>')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	436 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	437 if (p[1] == '=') {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	438 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	439 t.kind = TOK.Unordered;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	440 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	441 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	442 t.kind = TOK.UorE;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	443 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	444 else if (c == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	445 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	446 t.kind = TOK.UorG;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	447 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	448 else {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	449 t.kind = TOK.UorGorE;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	450 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	451 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	452 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	453 case '>':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	454 if (p[1] == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	455 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	456 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	457 t.kind = TOK.UorL;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	458 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	459 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	460 t.kind = TOK.UorLorE;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	461 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	462 case '=':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	463 t.kind = TOK.NotEqual;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	464 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	465 default:
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	466 t.kind = TOK.Not;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	467 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	468 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	469 assert(0);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	470 case '.': /* . .[0-9] .. ... */
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	471 if (p[1] == '.')
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	472 {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	473 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	474 if (p[1] == '.') {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	475 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	476 t.kind = TOK.Ellipses;
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	477 }
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	478 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	479 t.kind = TOK.Slice;
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	480 }
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	481 else if (isdigit(p[1]))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	482 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	483 return scanReal(t);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	484 }
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	485 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	486 t.kind = TOK.Dot;
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	487 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	488 case '\|': /* \| \|\| \|= */
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	489 c = *++p;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	490 if (c == '=')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	491 t.kind = TOK.OrAssign;
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	492 else if (c == '\|')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	493 t.kind = TOK.OrLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	494 else {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	495 t.kind = TOK.OrBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	496 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	497 }
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	498 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	499 case '&': /* & && &= */
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	500 c = *++p;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	501 if (c == '=')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	502 t.kind = TOK.AndAssign;
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	503 else if (c == '&')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	504 t.kind = TOK.AndLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	505 else {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	506 t.kind = TOK.AndBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	507 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	508 }
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	509 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	510 case '+': /* + ++ += */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	511 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	512 if (c == '=')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	513 t.kind = TOK.PlusAssign;
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	514 else if (c == '+')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	515 t.kind = TOK.PlusPlus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	516 else {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	517 t.kind = TOK.Plus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	518 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	519 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	520 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	521 case '-': /* - -- -= */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	522 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	523 if (c == '=')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	524 t.kind = TOK.MinusAssign;
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	525 else if (c == '-')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	526 t.kind = TOK.MinusMinus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	527 else {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	528 t.kind = TOK.Minus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	529 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	530 }
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	531 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	532 case '=': /* = == */
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	533 if (p[1] == '=') {
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	534 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	535 t.kind = TOK.Equal;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	536 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	537 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	538 t.kind = TOK.Assign;
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	539 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	540 case '~': /* ~ ~= */
27 43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	541 if (p[1] == '=') {
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	542 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	543 t.kind = TOK.CatAssign;
27 43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	544 }
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	545 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	546 t.kind = TOK.Tilde;
27 43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	547 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	548 case '': / * = /
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	549 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	550 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	551 t.kind = TOK.MulAssign;
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	552 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	553 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	554 t.kind = TOK.Mul;
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	555 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	556 case '^': /* ^ ^= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	557 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	558 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	559 t.kind = TOK.XorAssign;
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	560 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	561 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	562 t.kind = TOK.Xor;
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	563 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	564 case '%': /* % %= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	565 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	566 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	567 t.kind = TOK.ModAssign;
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	568 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	569 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	570 t.kind = TOK.Mod;
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	571 goto Lcommon;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	572 // Single character tokens:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	573 case '(':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	574 t.kind = TOK.LParen;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	575 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	576 case ')':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	577 t.kind = TOK.RParen;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	578 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	579 case '[':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	580 t.kind = TOK.LBracket;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	581 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	582 case ']':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	583 t.kind = TOK.RBracket;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	584 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	585 case '{':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	586 t.kind = TOK.LBrace;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	587 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	588 case '}':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	589 t.kind = TOK.RBrace;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	590 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	591 case ':':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	592 t.kind = TOK.Colon;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	593 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	594 case ';':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	595 t.kind = TOK.Semicolon;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	596 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	597 case '?':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	598 t.kind = TOK.Question;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	599 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	600 case ',':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	601 t.kind = TOK.Comma;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	602 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	603 case '$':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	604 t.kind = TOK.Dollar;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	605 Lcommon:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	606 ++p;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	607 Lcommon2:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	608 t.end = p;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	609 return;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	610 case '#':
360 b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence(). aziz parents: 350 diff changeset	611 return scanSpecialTokenSequence(t);
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	612 default:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	613 }
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	614
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	615 // Check for EOF
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	616 if (isEOF(c))
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	617 {
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	618 assert(isEOF(p), ""~p);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	619 t.kind = TOK.EOF;
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	620 t.end = p;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	621 tail = &t;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	622 assert(t.start == t.end);
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	623 return;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	624 }
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	625
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	626 if (!isascii(c))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	627 {
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	628 c = decodeUTF8();
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	629 if (isUniAlpha(c))
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	630 goto Lidentifier;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	631 }
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	632
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	633 error(t.start, MID.IllegalCharacter, cast(dchar)c);
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	634
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	635 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	636 t.kind = TOK.Illegal;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	637 t.setWhitespaceFlag();
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	638 t.dchar_ = c;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	639 t.end = p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	640 return;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	641 }
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	642 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	643
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	644 template toUint(char[] T)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	645 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	646 static assert(0 < T.length && T.length <= 4);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	647 static if (T.length == 1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	648 const uint toUint = T[0];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	649 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	650 const uint toUint = (T[0] << ((T.length-1)*8)) \| toUint!(T[1..$]);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	651 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	652 static assert(toUint!("\xAA\xBB\xCC\xDD") == 0xAABBCCDD);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	653
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	654 // Can't use this yet due to a bug in DMD (bug id=1534).
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	655 template case_(char[] str, TOK tok, char[] label)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	656 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	657 const char[] case_ =
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	658 `case `~toUint!(str).stringof~`:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	659
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	660 goto `~label~`;`;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	661 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	662
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	663 template case_L4(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	664 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	665 const char[] case_L4 = case_!(str, tok, "Lcommon_4");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	666 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	667
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	668 template case_L3(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	669 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	670 const char[] case_L3 = case_!(str, tok, "Lcommon_3");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	671 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	672
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	673 template case_L2(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	674 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	675 const char[] case_L2 = case_!(str, tok, "Lcommon_2");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	676 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	677
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	678 template case_L1(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	679 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	680 const char[] case_L3 = case_!(str, tok, "Lcommon");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	681 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	682
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	683 public void scan_(ref Token t)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	684 in
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	685 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	686 assert(text.ptr <= p && p < end);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	687 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	688 out
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	689 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	690 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	691 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	692 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	693 body
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	694 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	695 // Scan whitespace.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	696 if (isspace(*p))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	697 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	698 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	699 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	700 {}
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	701 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	702
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	703 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	704 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	705 // Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	706 switch (*p)
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	707 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	708 case '\r':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	709 if (p[1] == '\n')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	710 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	711 case '\n':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	712 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	713 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	714 ++lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	715 setLineBegin(p);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	716 // this.newline = &t;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	717 t.kind = TOK.Newline;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	718 t.setWhitespaceFlag();
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	719 t.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	720 t.newline.oriLineNum = lineNum;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	721 t.newline.setLineNum = lineNum_hline;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	722 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	723 return;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	724 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	725 if (isUnicodeNewline(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	726 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	727 ++p; ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	728 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	729 }
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	730 }
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	731
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	732 uint c = *p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	733 assert(end - p != 0);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	734 switch (end - p)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	735 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	736 case 1:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	737 goto L1character;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	738 case 2:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	739 c <<= 8; c \|= p[1];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	740 goto L2characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	741 case 3:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	742 c <<= 8; c \|= p[1]; c <<= 8; c \|= p[2];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	743 goto L3characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	744 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	745 version(BigEndian)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	746 c = cast(uint)p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	747 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	748 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	749 c <<= 8; c \|= p[1]; c <<= 8; c \|= p[2]; c <<= 8; c \|= p[3];
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	750 /+
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	751 c = cast(uint)p;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	752 asm
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	753 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	754 mov EDX, c;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	755 bswap EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	756 mov c, EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	757 }
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	758 +/
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	759 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	760 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	761
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	762 // 4 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	763 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	764 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	765 case toUint!(">>>="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	766 t.kind = TOK.RShiftAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	767 goto Lcommon_4;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	768 case toUint!("!<>="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	769 t.kind = TOK.Unordered;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	770 Lcommon_4:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	771 p += 4;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	772 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	773 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	774 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	775 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	776
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	777 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	778 L3characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	779 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	780 // 3 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	781 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	782 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	783 case toUint!(">>="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	784 t.kind = TOK.RShiftAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	785 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	786 case toUint!(">>>"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	787 t.kind = TOK.URShift;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	788 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	789 case toUint!("<>="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	790 t.kind = TOK.LorEorG;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	791 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	792 case toUint!("<<="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	793 t.kind = TOK.LShiftAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	794 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	795 case toUint!("!<="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	796 t.kind = TOK.UorG;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	797 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	798 case toUint!("!>="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	799 t.kind = TOK.UorL;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	800 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	801 case toUint!("!<>"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	802 t.kind = TOK.UorE;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	803 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	804 case toUint!("..."):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	805 t.kind = TOK.Ellipses;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	806 Lcommon_3:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	807 p += 3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	808 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	809 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	810 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	811 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	812
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	813 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	814 L2characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	815 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	816 // 2 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	817 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	818 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	819 case toUint!("/+"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	820 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	821 return scanNestedComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	822 case toUint!("/*"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	823 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	824 return scanBlockComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	825 case toUint!("//"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	826 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	827 assert(*p == '/');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	828 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	829 isascii(*p) \|\| decodeUTF8();
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	830 t.kind = TOK.Comment;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	831 t.setWhitespaceFlag();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	832 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	833 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	834 case toUint!(">="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	835 t.kind = TOK.GreaterEqual;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	836 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	837 case toUint!(">>"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	838 t.kind = TOK.RShift;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	839 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	840 case toUint!("<<"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	841 t.kind = TOK.LShift;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	842 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	843 case toUint!("<="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	844 t.kind = TOK.LessEqual;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	845 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	846 case toUint!("<>"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	847 t.kind = TOK.LorG;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	848 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	849 case toUint!("!<"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	850 t.kind = TOK.UorGorE;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	851 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	852 case toUint!("!>"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	853 t.kind = TOK.UorLorE;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	854 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	855 case toUint!("!="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	856 t.kind = TOK.NotEqual;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	857 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	858 case toUint!(".."):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	859 t.kind = TOK.Slice;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	860 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	861 case toUint!("&&"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	862 t.kind = TOK.AndLogical;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	863 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	864 case toUint!("&="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	865 t.kind = TOK.AndAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	866 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	867 case toUint!("\|\|"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	868 t.kind = TOK.OrLogical;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	869 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	870 case toUint!("\|="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	871 t.kind = TOK.OrAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	872 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	873 case toUint!("++"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	874 t.kind = TOK.PlusPlus;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	875 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	876 case toUint!("+="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	877 t.kind = TOK.PlusAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	878 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	879 case toUint!("--"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	880 t.kind = TOK.MinusMinus;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	881 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	882 case toUint!("-="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	883 t.kind = TOK.MinusAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	884 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	885 case toUint!("=="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	886 t.kind = TOK.Equal;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	887 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	888 case toUint!("~="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	889 t.kind = TOK.CatAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	890 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	891 case toUint!("*="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	892 t.kind = TOK.MulAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	893 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	894 case toUint!("/="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	895 t.kind = TOK.DivAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	896 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	897 case toUint!("^="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	898 t.kind = TOK.XorAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	899 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	900 case toUint!("%="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	901 t.kind = TOK.ModAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	902 Lcommon_2:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	903 p += 2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	904 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	905 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	906 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	907 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	908
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	909 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	910 L1character:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	911 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	912 assert(p == c, Format("p={0},c={1}", p, cast(dchar)c));
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	913 // 1 character tokens.
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	914 // TODO: consider storing the token type in ptable.
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	915 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	916 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	917 case '\'':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	918 return scanCharacterLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	919 case '`':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	920 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	921 case '"':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	922 return scanNormalStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	923 case '\\':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	924 char[] buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	925 do
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	926 {
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	927 bool isBinary;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	928 c = scanEscapeSequence(isBinary);
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	929 if (isascii(c) \|\| isBinary)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	930 buffer ~= c;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	931 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	932 encodeUTF8(buffer, c);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	933 } while (*p == '\\')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	934 buffer ~= 0;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	935 t.kind = TOK.String;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	936 t.str = buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	937 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	938 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	939 case '<':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	940 t.kind = TOK.Greater;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	941 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	942 case '>':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	943 t.kind = TOK.Less;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	944 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	945 case '^':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	946 t.kind = TOK.Xor;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	947 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	948 case '!':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	949 t.kind = TOK.Not;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	950 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	951 case '.':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	952 if (isdigit(p[1]))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	953 return scanReal(t);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	954 t.kind = TOK.Dot;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	955 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	956 case '&':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	957 t.kind = TOK.AndBinary;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	958 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	959 case '\|':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	960 t.kind = TOK.OrBinary;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	961 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	962 case '+':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	963 t.kind = TOK.Plus;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	964 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	965 case '-':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	966 t.kind = TOK.Minus;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	967 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	968 case '=':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	969 t.kind = TOK.Assign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	970 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	971 case '~':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	972 t.kind = TOK.Tilde;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	973 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	974 case '*':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	975 t.kind = TOK.Mul;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	976 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	977 case '/':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	978 t.kind = TOK.Div;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	979 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	980 case '%':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	981 t.kind = TOK.Mod;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	982 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	983 case '(':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	984 t.kind = TOK.LParen;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	985 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	986 case ')':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	987 t.kind = TOK.RParen;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	988 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	989 case '[':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	990 t.kind = TOK.LBracket;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	991 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	992 case ']':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	993 t.kind = TOK.RBracket;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	994 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	995 case '{':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	996 t.kind = TOK.LBrace;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	997 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	998 case '}':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	999 t.kind = TOK.RBrace;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1000 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1001 case ':':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1002 t.kind = TOK.Colon;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1003 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1004 case ';':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1005 t.kind = TOK.Semicolon;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1006 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1007 case '?':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1008 t.kind = TOK.Question;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1009 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1010 case ',':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1011 t.kind = TOK.Comma;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1012 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1013 case '$':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1014 t.kind = TOK.Dollar;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1015 Lcommon:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1016 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1017 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1018 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1019 case '#':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1020 return scanSpecialTokenSequence(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1021 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1022 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1023
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1024 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1025 assert(*p == c);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1026
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1027 // TODO: consider moving isidbeg() and isdigit() up.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1028 if (isidbeg(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1029 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1030 if (c == 'r' && p[1] == '"' && ++p)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1031 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1032 if (c == 'x' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1033 return scanHexStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1034 version(D2)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1035 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1036 if (c == 'q' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1037 return scanDelimitedStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1038 if (c == 'q' && p[1] == '{')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1039 return scanTokenStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1040 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1041 // Scan identifier.
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1042 Lidentifier:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1043 do
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1044 { c = *++p; }
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	1045 while (isident(c) \|\| !isascii(c) && isUnicodeAlpha())
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1046
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1047 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1048
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 499 diff changeset	1049 auto id = IdTable.lookup(t.srcText);
769 5e3ef1b2011c Added and improved documentation. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 764 diff changeset	1050 t.kind = id.kind;
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 499 diff changeset	1051 t.ident = id;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1052
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1053 if (t.kind == TOK.Identifier \|\| t.isKeyword)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1054 return;
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1055 else if (t.isSpecialToken)
0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1056 finalizeSpecialToken(t);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1057 else if (t.kind == TOK.EOF)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1058 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1059 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1060 assert(t.srcText == "__EOF__");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1061 }
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1062 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1063 assert(0, "unexpected token type: " ~ Token.toString(t.kind));
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1064 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1065 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1066
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1067 if (isdigit(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1068 return scanNumber(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1069
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1070 // Check for EOF
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1071 if (isEOF(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1072 {
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1073 assert(isEOF(p), p~"");
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1074 t.kind = TOK.EOF;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1075 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1076 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1077 assert(t.start == t.end);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1078 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1079 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1080
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1081 if (!isascii(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1082 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1083 c = decodeUTF8();
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1084 if (isUniAlpha(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1085 goto Lidentifier;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1086 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1087
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1088 error(t.start, MID.IllegalCharacter, cast(dchar)c);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1089
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1090 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1091 t.kind = TOK.Illegal;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	1092 t.setWhitespaceFlag();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1093 t.dchar_ = c;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1094 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1095 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1096 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1097
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1098 void scanBlockComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1099 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1100 assert(p[-1] == '/' && p == '');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1101 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1102 auto tokenLineBegin = lineBegin;
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1103 Loop:
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1104 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1105 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1106 switch (*++p)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1107 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1108 case '*':
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1109 if (p[1] != '/')
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1110 continue;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1111 p += 2;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1112 break Loop;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1113 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1114 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1115 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1116 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1117 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1118 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1119 setLineBegin(p+1);
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1120 break;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1121 default:
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1122 if (!isascii(*p))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1123 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1124 if (isUnicodeNewlineChar(decodeUTF8()))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1125 goto case '\n';
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1126 }
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1127 else if (isEOF(*p))
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1128 {
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1129 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedBlockComment);
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1130 break Loop;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1131 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1132 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1133 }
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1134 t.kind = TOK.Comment;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	1135 t.setWhitespaceFlag();
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1136 t.end = p;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1137 return;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1138 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1139
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1140 void scanNestedComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1141 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1142 assert(p[-1] == '/' && *p == '+');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1143 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1144 auto tokenLineBegin = lineBegin;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1145 uint level = 1;
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1146 Loop:
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1147 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1148 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1149 switch (*++p)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1150 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1151 case '/':
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1152 if (p[1] == '+')
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1153 ++p, ++level;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1154 continue;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1155 case '+':
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1156 if (p[1] != '/')
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1157 continue;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1158 ++p;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1159 if (--level != 0)
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1160 continue;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1161 ++p;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1162 break Loop;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1163 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1164 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1165 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1166 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1167 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1168 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1169 setLineBegin(p+1);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1170 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1171 default:
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1172 if (!isascii(*p))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1173 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1174 if (isUnicodeNewlineChar(decodeUTF8()))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1175 goto case '\n';
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1176 }
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1177 else if (isEOF(*p))
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1178 {
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1179 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedNestedComment);
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1180 break Loop;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1181 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1182 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1183 }
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1184 t.kind = TOK.Comment;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	1185 t.setWhitespaceFlag();
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1186 t.end = p;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1187 return;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1188 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1189
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1190 char scanPostfix()
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1191 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1192 assert(p[-1] == '"' \|\| p[-1] == '`' \|\|
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1193 { version(D2) return p[-1] == '}';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1194 else return 0; }()
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1195 );
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1196 switch (*p)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1197 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1198 case 'c':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1199 case 'w':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1200 case 'd':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1201 return *p++;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1202 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1203 return 0;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1204 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1205 assert(0);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1206 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1207
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1208 void scanNormalStringLiteral(ref Token t)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1209 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1210 assert(*p == '"');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1211 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1212 auto tokenLineBegin = lineBegin;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1213 t.kind = TOK.String;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1214 char[] buffer;
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1215 uint c;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1216 while (1)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1217 {
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1218 c = *++p;
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1219 switch (c)
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1220 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1221 case '"':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1222 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1223 t.pf = scanPostfix();
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1224 Lreturn:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1225 t.str = buffer ~ '\0';
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1226 t.end = p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1227 return;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1228 case '\\':
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1229 bool isBinary;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1230 c = scanEscapeSequence(isBinary);
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1231 --p;
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1232 if (isascii(c) \|\| isBinary)
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1233 buffer ~= c;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1234 else
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1235 encodeUTF8(buffer, c);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1236 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1237 case '\r':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1238 if (p[1] == '\n')
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1239 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1240 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1241 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1242 c = '\n'; // Convert Newline to \n.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1243 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1244 setLineBegin(p+1);
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1245 break;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1246 case 0, _Z_:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1247 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedString);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1248 goto Lreturn;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1249 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1250 if (!isascii(c))
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1251 {
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1252 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1253 if (isUnicodeNewlineChar(c))
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1254 goto case '\n';
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1255 encodeUTF8(buffer, c);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1256 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1257 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1258 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1259 assert(isascii(c));
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1260 buffer ~= c;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1261 }
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1262 assert(0);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1263 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1264
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1265 void scanCharacterLiteral(ref Token t)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1266 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1267 assert(*p == '\'');
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1268 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1269 t.kind = TOK.CharLiteral;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1270 switch (*p)
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1271 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1272 case '\\':
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1273 bool notused;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1274 t.dchar_ = scanEscapeSequence(notused);
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1275 break;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1276 case '\'':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1277 error(t.start, MID.EmptyCharacterLiteral);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1278 break;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1279 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1280 if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1281 break;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1282 uint c = *p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1283 if (!isascii(c))
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1284 c = decodeUTF8();
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1285 t.dchar_ = c;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1286 ++p;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1287 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1288
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1289 if (*p == '\'')
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1290 ++p;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1291 else
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1292 error(t.start, MID.UnterminatedCharacterLiteral);
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1293 t.end = p;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1294 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1295
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1296 void scanRawStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1297 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1298 assert(p == '`' \|\| p == '"' && p[-1] == 'r');
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1299 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1300 auto tokenLineBegin = lineBegin;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1301 t.kind = TOK.String;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1302 uint delim = *p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1303 char[] buffer;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1304 uint c;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1305 while (1)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1306 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1307 c = *++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1308 switch (c)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1309 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1310 case '\r':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1311 if (p[1] == '\n')
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1312 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1313 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1314 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1315 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1316 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1317 setLineBegin(p+1);
52 f65a83c27638 - Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well. aziz parents: 51 diff changeset	1318 break;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1319 case '`':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1320 case '"':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1321 if (c == delim)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1322 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1323 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1324 t.pf = scanPostfix();
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1325 Lreturn:
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1326 t.str = buffer ~ '\0';
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1327 t.end = p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1328 return;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1329 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1330 break;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1331 case 0, _Z_:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1332 error(tokenLineNum, tokenLineBegin, t.start,
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1333 delim == 'r' ? MID.UnterminatedRawString : MID.UnterminatedBackQuoteString);
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1334 goto Lreturn;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1335 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1336 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1337 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1338 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1339 if (isUnicodeNewlineChar(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1340 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1341 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1342 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1343 }
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1344 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1345 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1346 buffer ~= c;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1347 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1348 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1349 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1350
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1351 void scanHexStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1352 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1353 assert(p[0] == 'x' && p[1] == '"');
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1354 t.kind = TOK.String;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1355
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1356 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1357 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1358
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1359 uint c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1360 ubyte[] buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1361 ubyte h; // hex number
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1362 uint n; // number of hex digits
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1363
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1364 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1365 assert(*p == '"');
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1366 while (1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1367 {
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1368 c = *++p;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1369 switch (c)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1370 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1371 case '"':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1372 if (n & 1)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1373 error(tokenLineNum, tokenLineBegin, t.start, MID.OddNumberOfDigitsInHexString);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1374 ++p;
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1375 t.pf = scanPostfix();
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1376 Lreturn:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1377 t.str = cast(string) (buffer ~= 0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1378 t.end = p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1379 return;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1380 case '\r':
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1381 if (p[1] == '\n')
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1382 ++p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1383 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1384 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1385 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1386 setLineBegin(p+1);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1387 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1388 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1389 if (ishexad(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1390 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1391 if (c <= '9')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1392 c -= '0';
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1393 else if (c <= 'F')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1394 c -= 'A' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1395 else
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1396 c -= 'a' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1397
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1398 if (n & 1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1399 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1400 h <<= 4;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1401 h \|= c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1402 buffer ~= h;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1403 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1404 else
112 004d98df65af - Implemented parseInterfaceDeclaration(). aziz parents: 105 diff changeset	1405 h = cast(ubyte)c;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1406 ++n;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1407 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1408 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1409 else if (isspace(c))
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1410 continue; // Skip spaces.
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 494 diff changeset	1411 else if (isEOF(c))
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1412 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1413 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedHexString);
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1414 t.pf = 0;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1415 goto Lreturn;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1416 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1417 else
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1418 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1419 auto errorAt = p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1420 if (!isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1421 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1422 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1423 if (isUnicodeNewlineChar(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1424 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1425 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1426 error(errorAt, MID.NonHexCharInHexString, cast(dchar)c);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1427 }
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1428 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1429 }
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1430 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1431 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1432
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1433 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1434 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1435 void scanDelimitedStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1436 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1437 assert(p[0] == 'q' && p[1] == '"');
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1438 t.kind = TOK.String;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1439
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1440 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1441 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1442
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1443 char[] buffer;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1444 dchar opening_delim = 0, // 0 if no nested delimiter or '[', '(', '<', '{'
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1445 closing_delim; // Will be ']', ')', '>', '},
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1446 // the first character of an identifier or
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1447 // any other Unicode/ASCII character.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1448 char[] str_delim; // Identifier delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1449 uint level = 1; // Counter for nestable delimiters.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1450
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1451 ++p; ++p; // Skip q"
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1452 uint c = *p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1453 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1454 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1455 case '(':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1456 opening_delim = c;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1457 closing_delim = ')'; // c + 1
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1458 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1459 case '[', '<', '{':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1460 opening_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1461 closing_delim = c + 2; // Get to closing counterpart. Feature of ASCII table.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1462 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1463 default:
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1464 dchar scanNewline()
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1465 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1466 switch (*p)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1467 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1468 case '\r':
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1469 if (p[1] == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1470 ++p;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1471 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1472 assert(isNewlineEnd(p));
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1473 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1474 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1475 setLineBegin(p);
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1476 return '\n';
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1477 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1478 if (isUnicodeNewline(p))
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1479 {
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1480 ++p; ++p;
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1481 goto case '\n';
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1482 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1483 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1484 return 0;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1485 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1486 // Skip leading newlines:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1487 while (scanNewline() != 0)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1488 {}
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1489 assert(!isNewline(p));
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1490
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1491 char* begin = p;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1492 c = *p;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1493 closing_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1494 // TODO: Check for non-printable characters?
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1495 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1496 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1497 closing_delim = decodeUTF8();
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1498 if (!isUniAlpha(closing_delim))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1499 break; // Not an identifier.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1500 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1501 else if (!isidbeg(c))
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1502 break; // Not an identifier.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1503
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1504 // Parse Identifier + EndOfLine
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1505 do
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1506 { c = *++p; }
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	1507 while (isident(c) \|\| !isascii(c) && isUnicodeAlpha())
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1508 // Store identifier
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1509 str_delim = begin[0..p-begin];
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1510 // Scan newline
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1511 if (scanNewline() == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1512 --p; // Go back one because of "c = *++p;" in main loop.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1513 else
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1514 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1515 // TODO: error(p, MID.ExpectedNewlineAfterIdentDelim);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1516 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1517 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1518
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1519 bool checkStringDelim(char* p)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1520 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1521 assert(str_delim.length != 0);
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1522 if (buffer[$-1] == '\n' && // Last character copied to buffer must be '\n'.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1523 end-p >= str_delim.length && // Check remaining length.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1524 p[0..str_delim.length] == str_delim) // Compare.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1525 return true;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1526 return false;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1527 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1528
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1529 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1530 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1531 c = *++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1532 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1533 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1534 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1535 if (p[1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1536 ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1537 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1538 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1539 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1540 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1541 setLineBegin(p+1);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1542 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1543 case 0, _Z_:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1544 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedDelimitedString);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1545 goto Lreturn3;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1546 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1547 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1548 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1549 auto begin = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1550 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1551 if (isUnicodeNewlineChar(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1552 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1553 if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1554 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1555 if (str_delim.length)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1556 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1557 if (checkStringDelim(begin))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1558 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1559 p = begin + str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1560 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1561 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1562 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1563 else
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1564 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1565 assert(level == 1);
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1566 --level;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1567 goto Lreturn;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1568 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1569 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1570 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1571 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1572 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1573 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1574 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1575 if (c == opening_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1576 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1577 else if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1578 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1579 if (str_delim.length)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1580 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1581 if (checkStringDelim(p))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1582 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1583 p += str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1584 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1585 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1586 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1587 else if (--level == 0)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1588 goto Lreturn;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1589 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1590 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1591 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1592 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1593 buffer ~= c;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1594 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1595 Lreturn: // Character delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1596 assert(c == closing_delim);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1597 assert(level == 0);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1598 ++p; // Skip closing delimiter.
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1599 Lreturn2: // String delimiter.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1600 if (*p == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1601 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1602 else
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1603 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1604 // TODO: error(p, MID.ExpectedDblQuoteAfterDelim, str_delim.length ? str_delim : closing_delim~"");
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1605 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1606
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1607 t.pf = scanPostfix();
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1608 Lreturn3: // Error.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1609 t.str = buffer ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1610 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1611 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1612
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1613 void scanTokenStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1614 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1615 assert(p[0] == 'q' && p[1] == '{');
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1616 t.kind = TOK.String;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1617
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1618 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1619 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1620
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1621 // A guard against changes to particular members:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1622 // this.lineNum_hline and this.errorPath
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1623 ++inTokenString;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1624
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1625 uint lineNum = this.lineNum;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1626 uint level = 1;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1627
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1628 ++p; ++p; // Skip q{
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1629
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1630 auto prev_t = &t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1631 Token* token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1632 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1633 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1634 token = new Token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1635 scan(*token);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1636 // Save the tokens in a doubly linked list.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1637 // Could be useful for various tools.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1638 token.prev = prev_t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1639 prev_t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1640 prev_t = token;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1641 switch (token.kind)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1642 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1643 case TOK.LBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1644 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1645 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1646 case TOK.RBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1647 if (--level == 0)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1648 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1649 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1650 t.next = null;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1651 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1652 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1653 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1654 case TOK.EOF:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1655 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedTokenString);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1656 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1657 t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1658 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1659 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1660 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1661 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1662 break; // Exit loop.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1663 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1664
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1665 assert(token.kind == TOK.RBrace \|\| token.kind == TOK.EOF);
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1666 assert(token.kind == TOK.RBrace && t.next is null \|\|
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1667 token.kind == TOK.EOF && t.next !is null);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1668
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1669 char[] buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1670 // token points to } or EOF
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1671 if (token.kind == TOK.EOF)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1672 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1673 t.end = token.start;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1674 buffer = t.srcText[2..$].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1675 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1676 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1677 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1678 // Assign to buffer before scanPostfix().
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1679 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1680 buffer = t.srcText[2..$-1].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1681 t.pf = scanPostfix();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1682 t.end = p; // Assign again because of postfix.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1683 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1684 // Convert newlines to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1685 if (lineNum != this.lineNum)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1686 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1687 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1688 uint i, j;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1689 for (; i < buffer.length; ++i)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1690 switch (buffer[i])
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1691 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1692 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1693 if (buffer[i+1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1694 ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1695 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1696 assert(isNewlineEnd(buffer.ptr + i));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1697 buffer[j++] = '\n'; // Convert Newline to '\n'.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1698 break;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1699 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1700 if (isUnicodeNewline(buffer.ptr + i))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1701 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1702 ++i; ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1703 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1704 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1705 buffer[j++] = buffer[i]; // Copy.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1706 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1707 buffer.length = j; // Adjust length.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1708 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1709 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1710 t.str = buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1711
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1712 --inTokenString;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1713 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1714 } // version(D2)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1715
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1716 dchar scanEscapeSequence(ref bool isBinary)
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1717 out(result)
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	1718 { assert(isValidChar(result)); }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1719 body
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1720 {
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1721 assert(*p == '\\');
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1722
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1723 auto sequenceStart = p; // Used for error reporting.
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1724
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1725 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1726 uint c = char2ev(*p);
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1727 if (c)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1728 {
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1729 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1730 return c;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1731 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1732
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1733 uint digits = 2;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1734
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1735 switch (*p)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1736 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1737 case 'x':
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1738 isBinary = true;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1739 case_Unicode:
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1740 assert(c == 0);
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1741 assert(digits == 2 \|\| digits == 4 \|\| digits == 8);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1742 while (1)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1743 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1744 ++p;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1745 if (ishexad(*p))
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1746 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1747 c *= 16;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1748 if (*p <= '9')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1749 c += *p - '0';
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1750 else if (*p <= 'F')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1751 c += *p - 'A' + 10;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1752 else
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1753 c += *p - 'a' + 10;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1754
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1755 if (--digits == 0)
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1756 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1757 ++p;
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	1758 if (isValidChar(c))
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1759 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1760
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1761 error(sequenceStart, MID.InvalidUnicodeEscapeSequence,
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1762 sequenceStart[0..p-sequenceStart]);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1763 break;
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1764 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1765 continue;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1766 }
432 3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 430 diff changeset	1767
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1768 error(sequenceStart, MID.InsufficientHexDigits,
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1769 sequenceStart[0..p-sequenceStart]);
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1770 break;
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1771 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1772 break;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1773 case 'u':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1774 digits = 4;
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1775 goto case_Unicode;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1776 case 'U':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1777 digits = 8;
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1778 goto case_Unicode;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1779 default:
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1780 if (isoctal(*p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1781 {
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1782 isBinary = true;
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1783 assert(c == 0);
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1784 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1785 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1786 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1787 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1788 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1789 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1790 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1791 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1792 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1793 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1794 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1795 ++p;
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1796 return c & 0xFF; // Return valid escape value.
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1797 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1798 else if(*p == '&')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1799 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1800 if (isalpha(*++p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1801 {
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1802 auto begin = p;
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1803 while (isalnum(*++p))
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1804 {}
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1805
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1806 if (*p == ';')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1807 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1808 // Pass entity excluding '&' and ';'.
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1809 c = entity2Unicode(begin[0..p - begin]);
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1810 ++p; // Skip ;
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1811 if (c != 0xFFFF)
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1812 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1813 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1814 error(sequenceStart, MID.UndefinedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1815 }
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1816 else
432 3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 430 diff changeset	1817 error(sequenceStart, MID.UnterminatedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1818 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1819 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1820 error(sequenceStart, MID.InvalidBeginHTMLEntity);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1821 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1822 else if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1823 error(sequenceStart, MID.UndefinedEscapeSequence,
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 494 diff changeset	1824 isEOF(*p) ? `\EOF` : `\NewLine`);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1825 else
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1826 {
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1827 char[] str = `\`;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1828 if (isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1829 str ~= *p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1830 else
387 ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1831 encodeUTF8(str, decodeUTF8());
ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1832 ++p;
ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1833 // TODO: check for unprintable character?
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1834 error(sequenceStart, MID.UndefinedEscapeSequence, str);
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1835 }
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1836 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1837 return REPLACEMENT_CHAR; // Error: return replacement character.
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1838 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1839
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1840 /*
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1841 IntegerLiteral:= (Dec\|Hex\|Bin\|Oct)Suffix?
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1842 Dec:= (0\|[1-9][0-9_]*)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1843 Hex:= 0[xX] HexDigits
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1844 Bin:= 0[bB][01_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1845 Oct:= 0[0-7_]+
68 7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	1846 Suffix:= (L[uU]?\|[uU]L?)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1847 HexDigits:= [0-9a-zA-Z_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1848
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1849 Invalid: "0b_", "0x_", "._"
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1850 */
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1851 void scanNumber(ref Token t)
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1852 {
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1853 ulong ulong_;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1854 bool overflow;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1855 bool isDecimal;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1856 size_t digits;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1857
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1858 if (*p != '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1859 goto LscanInteger;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1860 ++p; // skip zero
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1861 // check for xX bB ...
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1862 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1863 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1864 case 'x','X':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1865 goto LscanHex;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1866 case 'b','B':
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1867 goto LscanBinary;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1868 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1869 if (p[1] == 'i')
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1870 goto LscanReal; // 0Li
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1871 break; // 0L
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1872 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1873 if (p[1] == '.')
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1874 break; // 0..
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1875 // 0.
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1876 case 'i','f','F', // Imaginary and float literal suffixes.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1877 'e', 'E': // Float exponent.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1878 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1879 default:
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1880 if (*p == '_')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1881 goto LscanOctal; // 0_
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1882 else if (isdigit(*p))
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1883 {
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1884 if (p == '8' \|\| p == '9')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1885 goto Loctal_hasDecimalDigits; // 08 or 09
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1886 else
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1887 goto Loctal_enter_loop; // 0[0-7]
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1888 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1889 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1890
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1891 // Number 0
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1892 assert(p[-1] == '0');
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1893 assert(p != '_' && !isdigit(p));
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1894 assert(ulong_ == 0);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1895 isDecimal = true;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1896 goto Lfinalize;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1897
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1898 LscanInteger:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1899 assert(p != 0 && isdigit(p));
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1900 isDecimal = true;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1901 goto Lenter_loop_int;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1902 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1903 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1904 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1905 continue;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1906 if (!isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1907 break;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1908 Lenter_loop_int:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1909 if (ulong_ < ulong.max/10 \|\| (ulong_ == ulong.max/10 && *p <= '5'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1910 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1911 ulong_ *= 10;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1912 ulong_ += *p - '0';
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1913 continue;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1914 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1915 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1916 overflow = true;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1917 while (isdigit(*++p)) {}
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1918 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1919 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1920
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1921 // The number could be a float, so check overflow below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1922 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1923 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1924 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1925 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1926 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1927 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1928 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1929 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1930 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1931 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1932 goto LscanReal;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1933 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1934 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1935
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1936 if (overflow)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1937 error(t.start, MID.OverflowDecimalNumber);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1938
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1939 assert((isdigit(p[-1]) \|\| p[-1] == '_') && !isdigit(p) && p != '_');
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1940 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1941
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1942 LscanHex:
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1943 assert(digits == 0);
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	1944 assert(p == 'x' \|\| p == 'X');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1945 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1946 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1947 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1948 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1949 if (!ishexad(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1950 break;
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1951 ++digits;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1952 ulong_ *= 16;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1953 if (*p <= '9')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1954 ulong_ += *p - '0';
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1955 else if (*p <= 'F')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1956 ulong_ += *p - 'A' + 10;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1957 else
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1958 ulong_ += *p - 'a' + 10;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1959 }
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1960
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	1961 assert(ishexad(p[-1]) \|\| p[-1] == '_' \|\| p[-1] == 'x' \|\| p[-1] == 'X');
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1962 assert(!ishexad(p) && p != '_');
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1963
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1964 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1965 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1966 case '.':
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1967 if (p[1] == '.')
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1968 break;
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1969 case 'p', 'P':
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1970 return scanHexReal(t);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1971 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	1972 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1973
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1974 if (digits == 0 \|\| digits > 16)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1975 error(t.start, digits == 0 ? MID.NoDigitsInHexNumber : MID.OverflowHexNumber);
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	1976
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1977 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1978
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1979 LscanBinary:
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1980 assert(digits == 0);
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	1981 assert(p == 'b' \|\| p == 'B');
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1982 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1983 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1984 if (*++p == '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1985 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1986 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1987 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1988 }
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1989 else if (*p == '1')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1990 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1991 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1992 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1993 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1994 }
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1995 else if (*p == '_')
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1996 continue;
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1997 else
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	1998 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1999 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2000
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2001 if (digits == 0 \|\| digits > 64)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2002 error(t.start, digits == 0 ? MID.NoDigitsInBinNumber : MID.OverflowBinaryNumber);
59 3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	2003
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	2004 assert(p[-1] == '0' \|\| p[-1] == '1' \|\| p[-1] == '_' \|\| p[-1] == 'b' \|\| p[-1] == 'B', p[-1] ~ "");
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2005 assert( !(p == '0' \|\| p == '1' \|\| *p == '_') );
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2006 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2007
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2008 LscanOctal:
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2009 assert(*p == '_');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2010 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2011 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2012 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2013 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2014 if (!isoctal(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2015 break;
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2016 Loctal_enter_loop:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2017 if (ulong_ < ulong.max/2 \|\| (ulong_ == ulong.max/2 && *p <= '1'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2018 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2019 ulong_ *= 8;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2020 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2021 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2022 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2023 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2024 overflow = true;
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2025 while (isoctal(*++p)) {}
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2026 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2027 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2028
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2029 bool hasDecimalDigits;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2030 if (isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2031 {
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2032 Loctal_hasDecimalDigits:
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2033 hasDecimalDigits = true;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2034 while (isdigit(*++p)) {}
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2035 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2036
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2037 // The number could be a float, so check errors below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2038 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2039 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2040 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2041 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2042 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2043 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2044 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2045 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2046 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2047 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2048 goto LscanReal;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2049 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2050 }
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2051
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2052 if (hasDecimalDigits)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2053 error(t.start, MID.OctalNumberHasDecimals);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2054
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2055 if (overflow)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2056 error(t.start, MID.OverflowOctalNumber);
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2057 // goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2058
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2059 Lfinalize:
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2060 enum Suffix
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2061 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2062 None = 0,
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2063 Unsigned = 1,
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2064 Long = 2
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2065 }
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2066
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2067 // Scan optional suffix: L, Lu, LU, u, uL, U or UL.
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2068 Suffix suffix;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2069 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2070 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2071 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2072 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2073 case 'L':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2074 if (suffix & Suffix.Long)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2075 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2076 suffix \|= Suffix.Long;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2077 ++p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2078 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2079 case 'u', 'U':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2080 if (suffix & Suffix.Unsigned)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2081 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2082 suffix \|= Suffix.Unsigned;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2083 ++p;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2084 continue;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2085 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2086 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2087 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2088 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2089 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2090
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2091 // Determine type of Integer.
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2092 switch (suffix)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2093 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2094 case Suffix.None:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2095 if (ulong_ & 0x8000_0000_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2096 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2097 if (isDecimal)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2098 error(t.start, MID.OverflowDecimalSign);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2099 t.kind = TOK.Uint64;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2100 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2101 else if (ulong_ & 0xFFFF_FFFF_0000_0000)
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2102 t.kind = TOK.Int64;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2103 else if (ulong_ & 0x8000_0000)
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2104 t.kind = isDecimal ? TOK.Int64 : TOK.Uint32;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2105 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2106 t.kind = TOK.Int32;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2107 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2108 case Suffix.Unsigned:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2109 if (ulong_ & 0xFFFF_FFFF_0000_0000)
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2110 t.kind = TOK.Uint64;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2111 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2112 t.kind = TOK.Uint32;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2113 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2114 case Suffix.Long:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2115 if (ulong_ & 0x8000_0000_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2116 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2117 if (isDecimal)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2118 error(t.start, MID.OverflowDecimalSign);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2119 t.kind = TOK.Uint64;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2120 }
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2121 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2122 t.kind = TOK.Int64;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2123 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2124 case Suffix.Unsigned \| Suffix.Long:
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2125 t.kind = TOK.Uint64;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2126 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2127 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2128 assert(0);
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2129 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2130 t.ulong_ = ulong_;
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	2131 t.end = p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2132 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2133 LscanReal:
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2134 scanReal(t);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2135 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2136 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2137
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2138 /*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2139 FloatLiteral:= Float[fFL]?i?
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2140 Float:= DecFloat \| HexFloat
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2141 DecFloat:= ([0-9][0-9_][.][0-9_]DecExponent?) \| [.][0-9][0-9_]DecExponent? \| [0-9][0-9_]DecExponent
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2142 DecExponent:= [eE][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2143 HexFloat:= 0[xX](HexDigits[.]HexDigits \| [.][0-9a-zA-Z]HexDigits? \| HexDigits)HexExponent
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2144 HexExponent:= [pP][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2145 */
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2146 void scanReal(ref Token t)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2147 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2148 if (*p == '.')
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2149 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2150 assert(p[1] != '.');
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2151 // This function was called by scan() or scanNumber().
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2152 while (isdigit(++p) \|\| p == '_') {}
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2153 }
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2154 else
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2155 // This function was called by scanNumber().
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2156 assert(delegate ()
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2157 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2158 switch (*p)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2159 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2160 case 'L':
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2161 if (p[1] != 'i')
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2162 return false;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2163 case 'i', 'f', 'F', 'e', 'E':
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2164 return true;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2165 default:
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2166 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2167 return false;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2168 }()
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2169 );
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2170
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2171 // Scan exponent.
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2172 if (p == 'e' \|\| p == 'E')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2173 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2174 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2175 if (p == '-' \|\| p == '+')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2176 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2177 if (isdigit(*p))
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2178 while (isdigit(++p) \|\| p == '_') {}
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2179 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2180 error(t.start, MID.FloatExpMustStartWithDigit);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2181 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2182
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2183 // Copy whole number and remove underscores from buffer.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2184 char[] buffer = t.start[0..p-t.start].dup;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2185 uint j;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2186 foreach (c; buffer)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2187 if (c != '_')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2188 buffer[j++] = c;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2189 buffer.length = j; // Adjust length.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2190 buffer ~= 0; // Terminate for C functions.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2191
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2192 finalizeFloat(t, buffer);
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2193 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2194
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2195 void scanHexReal(ref Token t)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2196 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2197 assert(p == '.' \|\| p == 'p' \|\| *p == 'P');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2198 MID mid;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2199 if (*p == '.')
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2200 while (ishexad(++p) \|\| p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2201 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2202 // Decimal exponent is required.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2203 if (p != 'p' && p != 'P')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2204 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2205 mid = MID.HexFloatExponentRequired;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2206 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2207 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2208 // Scan exponent
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2209 assert(p == 'p' \|\| p == 'P');
393 fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2210 ++p;
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2211 if (p == '+' \|\| p == '-')
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2212 ++p;
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2213 if (!isdigit(*p))
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2214 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2215 mid = MID.HexFloatExpMustStartWithDigit;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2216 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2217 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2218 while (isdigit(++p) \|\| p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2219 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2220 // Copy whole number and remove underscores from buffer.
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2221 char[] buffer = t.start[0..p-t.start].dup;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2222 uint j;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2223 foreach (c; buffer)
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2224 if (c != '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2225 buffer[j++] = c;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2226 buffer.length = j; // Adjust length.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2227 buffer ~= 0; // Terminate for C functions.
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2228 finalizeFloat(t, buffer);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2229 return;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2230 Lerr:
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2231 t.kind = TOK.Float32;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2232 t.end = p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2233 error(t.start, mid);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2234 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2235
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2236 void finalizeFloat(ref Token t, string buffer)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2237 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2238 assert(buffer[$-1] == 0);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2239 // Float number is well-formed. Check suffixes and do conversion.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2240 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2241 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2242 case 'f', 'F':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2243 t.kind = TOK.Float32;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2244 t.float_ = strtof(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2245 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2246 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2247 case 'L':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2248 t.kind = TOK.Float80;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2249 t.real_ = strtold(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2250 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2251 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2252 default:
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2253 t.kind = TOK.Float64;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2254 t.double_ = strtod(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2255 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2256 if (*p == 'i')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2257 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2258 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2259 t.kind += 3; // Switch to imaginary counterpart.
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2260 assert(t.kind == TOK.Imaginary32 \|\|
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2261 t.kind == TOK.Imaginary64 \|\|
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2262 t.kind == TOK.Imaginary80);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2263 }
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	2264 if (errno() == ERANGE)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2265 error(t.start, MID.OverflowFloatNumber);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2266 t.end = p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2267 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2268
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2269 /// Scan special token: #line Integer [Filespec] EndOfLine
360 b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence(). aziz parents: 350 diff changeset	2270 void scanSpecialTokenSequence(ref Token t)
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2271 {
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	2272 assert(*p == '#');
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2273 t.kind = TOK.HashLine;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	2274 t.setWhitespaceFlag();
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2275
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2276 MID mid;
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2277 char* errorAtColumn = p;
763 f26f13b5a3a3 Fixed code in scanSpecialTokenSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 762 diff changeset	2278 char* tokenEnd = ++p;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2279
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2280 if (!(p[0] == 'l' && p[1] == 'i' && p[2] == 'n' && p[3] == 'e'))
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2281 {
370 ae4afb66768f - Renamed findModule() to findModulePath(). aziz parents: 367 diff changeset	2282 mid = MID.ExpectedIdentifierSTLine;
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2283 goto Lerr;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2284 }
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2285 p += 3;
763 f26f13b5a3a3 Fixed code in scanSpecialTokenSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 762 diff changeset	2286 tokenEnd = p + 1;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2287
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2288 // TODO: #line58"path/file" is legal. Require spaces?
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2289 // State.Space could be used for that purpose.
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2290 enum State
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2291 { /+Space,+/ Integer, Filespec, End }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2292
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2293 State state = State.Integer;
763 f26f13b5a3a3 Fixed code in scanSpecialTokenSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 762 diff changeset	2294
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2295 while (!isEndOfLine(++p))
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2296 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2297 if (isspace(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2298 continue;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2299 if (state == State.Integer)
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2300 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2301 if (!isdigit(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2302 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2303 errorAtColumn = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2304 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2305 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2306 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2307 t.tokLineNum = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2308 scan(*t.tokLineNum);
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2309 tokenEnd = p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2310 if (t.tokLineNum.kind != TOK.Int32 && t.tokLineNum.kind != TOK.Uint32)
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2311 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2312 errorAtColumn = t.tokLineNum.start;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2313 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2314 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2315 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2316 --p; // Go one back because scan() advanced p past the integer.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2317 state = State.Filespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2318 }
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2319 else if (state == State.Filespec && *p == '"')
763 f26f13b5a3a3 Fixed code in scanSpecialTokenSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 762 diff changeset	2320 { // MID.ExpectedFilespec is deprecated.
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2321 // if (*p != '"')
c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2322 // {
c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2323 // errorAtColumn = p;
c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2324 // mid = MID.ExpectedFilespec;
c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2325 // goto Lerr;
c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2326 // }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2327 t.tokLineFilespec = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2328 t.tokLineFilespec.start = p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2329 t.tokLineFilespec.kind = TOK.Filespec;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	2330 t.tokLineFilespec.setWhitespaceFlag();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2331 while (*++p != '"')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2332 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2333 if (isEndOfLine(p))
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2334 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2335 errorAtColumn = t.tokLineFilespec.start;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2336 mid = MID.UnterminatedFilespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2337 t.tokLineFilespec.end = p;
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2338 tokenEnd = p;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2339 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2340 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2341 isascii(*p) \|\| decodeUTF8();
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2342 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2343 auto start = t.tokLineFilespec.start +1; // +1 skips '"'
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2344 t.tokLineFilespec.str = start[0 .. p - start];
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2345 t.tokLineFilespec.end = p + 1;
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2346 tokenEnd = p + 1;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2347 state = State.End;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2348 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2349 else/+ if (state == State.End)+/
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2350 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2351 mid = MID.UnterminatedSpecialToken;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2352 goto Lerr;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2353 }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2354 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2355 assert(isEndOfLine(p));
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2356
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2357 if (state == State.Integer)
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2358 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2359 errorAtColumn = p;
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2360 mid = MID.ExpectedIntegerAfterSTLine;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2361 goto Lerr;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2362 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2363
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2364 // Evaluate #line only when not in token string.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2365 if (!inTokenString && t.tokLineNum)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2366 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2367 this.lineNum_hline = this.lineNum - t.tokLineNum.uint_ + 1;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2368 if (t.tokLineFilespec)
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	2369 newFilePath(t.tokLineFilespec.str);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2370 }
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2371 p = tokenEnd;
c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2372 t.end = tokenEnd;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2373
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2374 return;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2375 Lerr:
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2376 p = tokenEnd;
c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2377 t.end = tokenEnd;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2378 error(errorAtColumn, mid);
371 01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	2379 }
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	2380
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2381 /++
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2382 Insert an empty dummy token before t.
737 f88b5285b86b Implemented DDocEmitter. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 715 diff changeset	2383
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2384 Useful in the parsing phase for representing a node in the AST
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2385 that doesn't consume an actual token from the source text.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2386 +/
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2387 Token* insertEmptyTokenBefore(Token* t)
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2388 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2389 assert(t !is null && t.prev !is null);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2390 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2391 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2392
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2393 auto prev_t = t.prev;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2394 auto new_t = new Token;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2395 new_t.kind = TOK.Empty;
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2396 new_t.start = new_t.end = prev_t.end;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2397 // Link in new token.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2398 prev_t.next = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2399 new_t.prev = prev_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2400 new_t.next = t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2401 t.prev = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2402 return new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2403 }
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	2404
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2405 uint errorLineNumber(uint lineNum)
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	2406 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2407 return lineNum - this.lineNum_hline;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	2408 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	2409
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2410 void error(char* columnPos, MID mid, ...)
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2411 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2412 error_(this.lineNum, this.lineBegin, columnPos, mid, _arguments, _argptr);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2413 }
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2414
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2415 void error(uint lineNum, char* lineBegin, char* columnPos, MID mid, ...)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2416 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2417 error_(lineNum, lineBegin, columnPos, mid, _arguments, _argptr);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2418 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2419
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2420 void error_(uint lineNum, char* lineBegin, char* columnPos, MID mid,
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2421 TypeInfo[] _arguments, Arg _argptr)
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2422 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2423 lineNum = this.errorLineNumber(lineNum);
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	2424 auto errorPath = this.filePaths.setPath;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2425 auto location = new Location(errorPath, lineNum, lineBegin, columnPos);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2426 auto msg = Format(_arguments, _argptr, GetMsg(mid));
520 f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	2427 auto error = new LexerError(location, msg);
f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	2428 errors ~= error;
f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	2429 if (infoMan !is null)
f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	2430 infoMan ~= error;
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2431 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2432
306 9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2433 Token* getTokens()
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	2434 {
306 9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2435 while (nextToken() != TOK.EOF)
9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2436 {}
9c866aadcb5b - Moved code out of main() to separate functions. aziz parents: 272 diff changeset	2437 return head;
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	2438 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2439
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2440 /// Scan the whole text until EOF is encountered.
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2441 void scanAll()
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2442 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2443 while (nextToken() != TOK.EOF)
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2444 {}
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2445 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2446
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2447 /// HEAD -> Newline -> First Token
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2448 Token* firstToken()
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2449 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2450 return this.head.next.next;
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2451 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2452
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2453 static void loadKeywords(ref Identifier[string] table)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2454 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2455 foreach(k; keywords)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2456 table[k.str] = k;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2457 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2458
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2459 /// Returns true if str is a valid D identifier.
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2460 static bool isIdentifierString(char[] str)
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2461 {
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2462 if (str.length == 0 \|\| isdigit(str[0]))
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2463 return false;
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2464 size_t idx;
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2465 do
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2466 {
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2467 auto c = dil.Unicode.decode(str, idx);
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2468 if (c == ERROR_CHAR \|\| !(isident(c) \|\| !isascii(c) && isUniAlpha(c)))
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2469 return false;
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2470 } while (idx < str.length)
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2471 return true;
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2472 }
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2473
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2474 /// Returns true if str is a keyword or a special token (__FILE__, __LINE__ etc.)
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2475 static bool isReservedIdentifier(char[] str)
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2476 {
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2477 if (str.length == 0)
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2478 return false;
66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2479
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2480 static Identifier[string] reserved_ids_table;
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2481 if (reserved_ids_table is null)
499 52447db67938 Implemented global table of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 498 diff changeset	2482 Lexer.loadKeywords(reserved_ids_table);
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2483
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2484 if (!isIdentifierString(str))
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2485 return false;
ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2486
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2487 return (str in reserved_ids_table) !is null;
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2488 }
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2489
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2490 /++
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2491 Returns true if the current character to be decoded is
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2492 a Unicode alpha character.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2493 The current pointer 'p' is not advanced if false is returned.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2494 +/
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2495 bool isUnicodeAlpha()
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2496 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2497 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2498 char* p = this.p;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2499 dchar d = *p;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2500 ++p; // Move to second byte.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2501 // Error if second byte is not a trail byte.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2502 if (!isTrailByte(*p))
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2503 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2504 // Check for overlong sequences.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2505 switch (d)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2506 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2507 case 0xE0, 0xF0, 0xF8, 0xFC:
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2508 if ((*p & d) == 0x80)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2509 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2510 default:
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2511 if ((d & 0xFE) == 0xC0) // 1100000x
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2512 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2513 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2514 const char[] checkNextByte = "if (!isTrailByte(*++p))"
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2515 " return false;";
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2516 const char[] appendSixBits = "d = (d << 6) \| *p & 0b0011_1111;";
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2517 // Decode
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2518 if ((d & 0b1110_0000) == 0b1100_0000)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2519 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2520 d &= 0b0001_1111;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2521 mixin(appendSixBits);
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2522 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2523 else if ((d & 0b1111_0000) == 0b1110_0000)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2524 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2525 d &= 0b0000_1111;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2526 mixin(appendSixBits ~
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2527 checkNextByte ~ appendSixBits);
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2528 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2529 else if ((d & 0b1111_1000) == 0b1111_0000)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2530 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2531 d &= 0b0000_0111;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2532 mixin(appendSixBits ~
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2533 checkNextByte ~ appendSixBits ~
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2534 checkNextByte ~ appendSixBits);
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2535 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2536 else
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2537 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2538
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2539 assert(isTrailByte(*p));
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2540 if (!isValidChar(d) \|\| !isUniAlpha(d))
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2541 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2542 // Only advance pointer if this is a Unicode alpha character.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2543 this.p = p;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2544 return true;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2545 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2546
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2547 /// Decodes the next UTF-8 sequence.
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2548 dchar decodeUTF8()
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2549 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2550 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2551 char* p = this.p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2552 dchar d = *p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2553
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2554 ++p; // Move to second byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2555 // Error if second byte is not a trail byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2556 if (!isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2557 goto Lerr2;
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2558
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2559 // Check for overlong sequences.
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2560 switch (d)
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2561 {
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2562 case 0xE0, // 11100000 100xxxxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2563 0xF0, // 11110000 1000xxxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2564 0xF8, // 11111000 10000xxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2565 0xFC: // 11111100 100000xx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2566 if ((*p & d) == 0x80)
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2567 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2568 default:
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2569 if ((d & 0xFE) == 0xC0) // 1100000x
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2570 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2571 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2572
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2573 const char[] checkNextByte = "if (!isTrailByte(*++p))"
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2574 " goto Lerr2;";
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2575 const char[] appendSixBits = "d = (d << 6) \| *p & 0b0011_1111;";
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2576
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2577 // Decode
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2578 if ((d & 0b1110_0000) == 0b1100_0000)
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2579 { // 110xxxxx 10xxxxxx
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2580 d &= 0b0001_1111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2581 mixin(appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2582 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2583 else if ((d & 0b1111_0000) == 0b1110_0000)
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2584 { // 1110xxxx 10xxxxxx 10xxxxxx
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2585 d &= 0b0000_1111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2586 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2587 checkNextByte ~ appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2588 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2589 else if ((d & 0b1111_1000) == 0b1111_0000)
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2590 { // 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2591 d &= 0b0000_0111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2592 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2593 checkNextByte ~ appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2594 checkNextByte ~ appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2595 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2596 else
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2597 // 5 and 6 byte UTF-8 sequences are not allowed yet.
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2598 // 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2599 // 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2600 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2601
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2602 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2603
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2604 if (!isValidChar(d))
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2605 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2606 Lerr:
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2607 // Three cases:
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2608 // *) the UTF-8 sequence was successfully decoded but the resulting
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2609 // character is invalid.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2610 // p points to last trail byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2611 // *) the UTF-8 sequence is overlong.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2612 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2613 // *) the UTF-8 sequence has more than 4 bytes or starts with
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2614 // a trail byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2615 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2616 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2617 // Move to next ASCII character or lead byte of a UTF-8 sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2618 while (p < (end-1) && isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2619 ++p;
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2620 --p;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2621 assert(!isTrailByte(p[1]));
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2622 Lerr2:
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2623 d = REPLACEMENT_CHAR;
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2624 error(this.p, MID.InvalidUTF8Sequence, formatBytes(this.p, p));
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2625 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2626
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2627 this.p = p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2628 return d;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2629 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2630
519 9ebc799c7dc5 Fixes in dil.Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 514 diff changeset	2631 static void encodeUTF8(ref char[] str, dchar d)
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2632 {
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2633 assert(!isascii(d), "check for ASCII char before calling encodeUTF8().");
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2634 assert(isValidChar(d), "check if character is valid before calling encodeUTF8().");
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2635
519 9ebc799c7dc5 Fixes in dil.Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 514 diff changeset	2636 char[6] b = void;
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2637 if (d < 0x800)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2638 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2639 b[0] = 0xC0 \| (d >> 6);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2640 b[1] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2641 str ~= b[0..2];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2642 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2643 else if (d < 0x10000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2644 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2645 b[0] = 0xE0 \| (d >> 12);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2646 b[1] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2647 b[2] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2648 str ~= b[0..3];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2649 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2650 else if (d < 0x200000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2651 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2652 b[0] = 0xF0 \| (d >> 18);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2653 b[1] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2654 b[2] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2655 b[3] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2656 str ~= b[0..4];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2657 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2658 /+ // There are no 5 and 6 byte UTF-8 sequences yet.
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2659 else if (d < 0x4000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2660 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2661 b[0] = 0xF8 \| (d >> 24);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2662 b[1] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2663 b[2] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2664 b[3] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2665 b[4] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2666 str ~= b[0..5];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2667 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2668 else if (d < 0x80000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2669 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2670 b[0] = 0xFC \| (d >> 30);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2671 b[1] = 0x80 \| ((d >> 24) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2672 b[2] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2673 b[3] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2674 b[4] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2675 b[5] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2676 str ~= b[0..6];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2677 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2678 +/
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2679 else
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2680 assert(0);
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2681 }
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2682
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2683 /// Formats the bytes between start and end.
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2684 /// Returns: e.g.: abc -> \x61\x62\x63
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2685 static char[] formatBytes(char* start, char* end)
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2686 {
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2687 auto strLen = end-start;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2688 const formatLen = `\xXX`.length;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2689 char[] result = new char[strLen*formatLen]; // Reserve space.
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2690 result.length = 0;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2691 foreach (c; cast(ubyte[])start[0..strLen])
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2692 result ~= Format("\\x{:X}", c);
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2693 return result;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2694 }
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2695
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2696 /// Searches for an invalid UTF-8 sequence in str.
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2697 /// Returns: a formatted string of the invalid sequence (e.g. \xC0\x80).
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2698 static string findInvalidUTF8Sequence(string str)
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2699 {
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2700 char* p = str.ptr, end = p + str.length;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2701 while (p < end)
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2702 {
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2703 if (decode(p, end) == ERROR_CHAR)
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2704 {
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2705 auto begin = p;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2706 // Skip trail-bytes.
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2707 while (++p < end && isTrailByte(*p))
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2708 {}
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2709 return Lexer.formatBytes(begin, p);
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2710 }
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2711 }
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2712 assert(p == end);
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2713 return "";
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2714 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	2715 }
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2716
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2717 unittest
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2718 {
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	2719 Stdout("Testing Lexer.\n");
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2720 struct Pair
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2721 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2722 char[] tokenText;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2723 TOK kind;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2724 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2725 static Pair[] pairs = [
494 9a7ca8c56e59 Refactored a few things in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	2726 {"#!äöüß", TOK.Shebang}, {"\n", TOK.Newline},
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2727 {"//çay", TOK.Comment}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2728 {"&", TOK.AndBinary},
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2729 {"/çağ/", TOK.Comment}, {"&&", TOK.AndLogical},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2730 {"/+çak+/", TOK.Comment}, {"&=", TOK.AndAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2731 {">", TOK.Greater}, {"+", TOK.Plus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2732 {">=", TOK.GreaterEqual}, {"++", TOK.PlusPlus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2733 {">>", TOK.RShift}, {"+=", TOK.PlusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2734 {">>=", TOK.RShiftAssign}, {"-", TOK.Minus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2735 {">>>", TOK.URShift}, {"--", TOK.MinusMinus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2736 {">>>=", TOK.URShiftAssign}, {"-=", TOK.MinusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2737 {"<", TOK.Less}, {"=", TOK.Assign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2738 {"<=", TOK.LessEqual}, {"==", TOK.Equal},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2739 {"<>", TOK.LorG}, {"~", TOK.Tilde},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2740 {"<>=", TOK.LorEorG}, {"~=", TOK.CatAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2741 {"<<", TOK.LShift}, {"*", TOK.Mul},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2742 {"<<=", TOK.LShiftAssign}, {"*=", TOK.MulAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2743 {"!", TOK.Not}, {"/", TOK.Div},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2744 {"!=", TOK.NotEqual}, {"/=", TOK.DivAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2745 {"!<", TOK.UorGorE}, {"^", TOK.Xor},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2746 {"!>", TOK.UorLorE}, {"^=", TOK.XorAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2747 {"!<=", TOK.UorG}, {"%", TOK.Mod},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2748 {"!>=", TOK.UorL}, {"%=", TOK.ModAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2749 {"!<>", TOK.UorE}, {"(", TOK.LParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2750 {"!<>=", TOK.Unordered}, {")", TOK.RParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2751 {".", TOK.Dot}, {"[", TOK.LBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2752 {"..", TOK.Slice}, {"]", TOK.RBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2753 {"...", TOK.Ellipses}, {"{", TOK.LBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2754 {"\|", TOK.OrBinary}, {"}", TOK.RBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2755 {"\|\|", TOK.OrLogical}, {":", TOK.Colon},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2756 {"\|=", TOK.OrAssign}, {";", TOK.Semicolon},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2757 {"?", TOK.Question}, {",", TOK.Comma},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2758 {"$", TOK.Dollar}, {"cam", TOK.Identifier},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2759 {"çay", TOK.Identifier}, {".0", TOK.Float64},
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2760 {"0", TOK.Int32}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2761 {"\r", TOK.Newline}, {"\r\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2762 {"\u2028", TOK.Newline}, {"\u2029", TOK.Newline}
40 9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2763 ];
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2764
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2765 char[] src;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2766
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2767 // Join all token texts into a single string.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2768 foreach (i, pair; pairs)
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2769 if (pair.kind == TOK.Comment && pair.tokenText[1] == '/' \|\| // Line comment.
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2770 pair.kind == TOK.Shebang)
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2771 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2772 assert(pairs[i+1].kind == TOK.Newline); // Must be followed by a newline.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2773 src ~= pair.tokenText;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2774 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2775 else
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2776 src ~= pair.tokenText ~ " ";
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2777
764 4579e8505d5e Fixed unittests and removed dil.File. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 763 diff changeset	2778 auto lx = new Lexer(new SourceText("", src));
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2779 auto token = lx.getTokens();
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2780
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2781 uint i;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2782 assert(token == lx.head);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2783 assert(token.next.kind == TOK.Newline);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2784 token = token.next.next;
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2785 do
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2786 {
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2787 assert(i < pairs.length);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2788 assert(token.srcText == pairs[i].tokenText, Format("Scanned '{0}' but expected '{1}'", token.srcText, pairs[i].tokenText));
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2789 ++i;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2790 token = token.next;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2791 } while (token.kind != TOK.EOF)
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	2792 }
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2793
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2794 unittest
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2795 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2796 Stdout("Testing method Lexer.peek()\n");
764 4579e8505d5e Fixed unittests and removed dil.File. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 763 diff changeset	2797 auto sourceText = new SourceText("", "unittest { }");
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2798 auto lx = new Lexer(sourceText, null);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2799
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2800 auto next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2801 lx.peek(next);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2802 assert(next.kind == TOK.Newline);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2803 lx.peek(next);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2804 assert(next.kind == TOK.Unittest);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2805 lx.peek(next);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2806 assert(next.kind == TOK.LBrace);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2807 lx.peek(next);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2808 assert(next.kind == TOK.RBrace);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2809 lx.peek(next);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2810 assert(next.kind == TOK.EOF);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2811
764 4579e8505d5e Fixed unittests and removed dil.File. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 763 diff changeset	2812 lx = new Lexer(new SourceText("", ""));
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2813 next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2814 lx.peek(next);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2815 assert(next.kind == TOK.Newline);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2816 lx.peek(next);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2817 assert(next.kind == TOK.EOF);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2818 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2819
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2820 unittest
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2821 {
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2822 // Numbers unittest
68 7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2823 // 0L 0ULi 0_L 0_UL 0x0U 0x0p2 0_Fi 0_e2 0_F 0_i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2824 // 0u 0U 0uL 0UL 0L 0LU 0Lu
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2825 // 0Li 0f 0F 0fi 0Fi 0i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2826 // 0b_1_LU 0b1000u
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2827 // 0x232Lu
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2828 }

Mercurial > projects > dil

annotate trunk/src/dil/lexer/Lexer.d @ 789:c1d5cfd7aa44