projects/dil: trunk/src/dil/lexer/Lexer.d annotate

annotate trunk/src/dil/lexer/Lexer.d @ 798:c24be8d4f6ab

Added documentation comments.

author	Aziz K?ksal <aziz.koeksal@gmail.com>
date	Sat, 01 Mar 2008 02:53:06 +0100
parents	cf2ad5df025c
children	cb8040538772

rev	line source
0 8ba2570de175 Initial import. aziz parents: diff changeset	1 /++
8ba2570de175 Initial import. aziz parents: diff changeset	2 Author: Aziz Köksal
249 32d354584b28 - Upgraded license notices to GPL3. aziz parents: 239 diff changeset	3 License: GPL3
0 8ba2570de175 Initial import. aziz parents: diff changeset	4 +/
576 0df647660e76 Moved Lexer.d to new package 'lexer'. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 552 diff changeset	5 module dil.lexer.Lexer;
577 9e811db780a6 Moved LexerFuncs.d to package 'lexer'. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 576 diff changeset	6
596 39fac5531b85 Moved dil.Token to dil.lexer.Token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 577 diff changeset	7 import dil.lexer.Token;
599 c4cdea3a65ba Moved dil.Keywords to dil.lexer.Keywords. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 596 diff changeset	8 import dil.lexer.Keywords;
600 041eae272362 Moved dil.Identifier to dil.lexer.Identifier. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 599 diff changeset	9 import dil.lexer.Identifier;
601 9f96fd51cb41 Moved dil.IdTable to dil.lexer.IdTable. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 600 diff changeset	10 import dil.lexer.IdTable;
327 a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	11 import dil.Information;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	12 import dil.Messages;
a48a987f7515 - Added package dil to import declarations. aziz parents: 326 diff changeset	13 import dil.HtmlEntities;
464 325714d8aa6c Added new module with compiler constants to resolve import cycle. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 434 diff changeset	14 import dil.CompilerInfo;
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	15 import dil.Unicode;
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	16 import dil.SourceText;
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	17 import dil.Time;
737 f88b5285b86b Implemented DDocEmitter. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 715 diff changeset	18 import common;
f88b5285b86b Implemented DDocEmitter. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 715 diff changeset	19
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	20 import tango.stdc.stdlib : strtof, strtod, strtold;
33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	21 import tango.stdc.errno : errno, ERANGE;
0 8ba2570de175 Initial import. aziz parents: diff changeset	22
577 9e811db780a6 Moved LexerFuncs.d to package 'lexer'. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 576 diff changeset	23 public import dil.lexer.Funcs;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	24
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	25 /// The Lexer analyzes the characters of a source text and
580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	26 /// produces a doubly-linked list of tokens.
0 8ba2570de175 Initial import. aziz parents: diff changeset	27 class Lexer
8ba2570de175 Initial import. aziz parents: diff changeset	28 {
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	29 SourceText srcText; /// The source text.
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	30 char* p; /// Points to the current character in the source text.
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	31 char* end; /// Points one character past the end of the source text.
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	32
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	33 Token* head; /// The head of the doubly linked token list.
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	34 Token* tail; /// The tail of the linked list. Set in scan().
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	35 Token* token; /// Points to the current token in the token list.
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	36
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	37 // Members used for error messages:
532 50e64bab9c7a Renamed InformationManager to InfoManager. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 520 diff changeset	38 InfoManager infoMan;
514 6ddff941862a Added new error classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 510 diff changeset	39 LexerError[] errors;
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	40 /// Always points to the first character of the current line.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	41 char* lineBegin;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	42 // Token* newline; /// Current newline token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	43 uint lineNum = 1; /// Current, actual source text line number.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	44 uint lineNum_hline; /// Line number set by #line.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	45 uint inTokenString; /// > 0 if inside q{ }
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	46 /// Holds the original file path and the modified one (by #line.)
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	47 NewlineData.FilePaths* filePaths;
103 511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too. aziz parents: 94 diff changeset	48
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	49 /// Construct a Lexer object.
580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	50 /// Params:
580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	51 /// srcText = the UTF-8 source code.
580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	52 /// infoMan = used for collecting error messages.
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	53 this(SourceText srcText, InfoManager infoMan = null)
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	54 {
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	55 this.srcText = srcText;
520 f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	56 this.infoMan = infoMan;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	57
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	58 assert(text.length && text[$-1] == 0, "source text has no sentinel character");
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	59 this.p = text.ptr;
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	60 this.end = this.p + text.length;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	61 this.lineBegin = this.p;
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	62
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	63 this.head = new Token;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	64 this.head.kind = TOK.HEAD;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	65 this.head.start = this.head.end = this.p;
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	66 this.token = this.head;
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	67 // Initialize this.filePaths.
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	68 newFilePath(this.srcText.filePath);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	69 // Add a newline as the first token after the head.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	70 auto newline = new Token;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	71 newline.kind = TOK.Newline;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	72 newline.setWhitespaceFlag();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	73 newline.start = newline.end = this.p;
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	74 newline.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	75 newline.newline.oriLineNum = 1;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	76 newline.newline.setLineNum = 0;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	77 // Link in.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	78 this.token.next = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	79 newline.prev = this.token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	80 this.token = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	81 // this.newline = newline;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	82 scanShebang();
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	83 }
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	84
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	85 /// The destructor deletes the doubly-linked token list.
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	86 ~this()
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	87 {
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	88 auto token = head.next;
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	89 while (token !is null)
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	90 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	91 assert(token.kind == TOK.EOF ? token == tail && token.next is null : 1);
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	92 delete token.prev;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	93 token = token.next;
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	94 }
344 757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	95 delete tail;
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	96 }
757c86e2c3cc - Added member tail and destructor method to Lexer. aziz parents: 343 diff changeset	97
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	98 char[] text()
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	99 {
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	100 return srcText.data;
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	101 }
90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	102
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	103 /// The "shebang" may optionally appear once at the beginning of a file.
580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	104 /// Regexp: #![^\EndOfLine]*
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	105 void scanShebang()
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	106 {
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	107 if (*p == '#' && p[1] == '!')
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	108 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	109 auto t = new Token;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	110 t.kind = TOK.Shebang;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	111 t.setWhitespaceFlag();
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	112 t.start = p;
29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	113 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	114 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	115 isascii(*p) \|\| decodeUTF8();
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	116 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	117 this.token.next = t;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	118 t.prev = this.token;
315 29c33ce6c5bb - Added method scanShebang to class Lexer. aziz parents: 309 diff changeset	119 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	120 }
92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	121
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	122 /// Sets the value of the special token.
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	123 void finalizeSpecialToken(ref Token t)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	124 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	125 assert(t.srcText[0..2] == "__");
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	126 switch (t.kind)
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	127 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	128 case TOK.FILE:
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	129 t.str = this.filePaths.setPath;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	130 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	131 case TOK.LINE:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	132 t.uint_ = this.errorLineNumber(this.lineNum);
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	133 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	134 case TOK.DATE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	135 TOK.TIME,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	136 TOK.TIMESTAMP:
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	137 auto time_str = Time.toString();
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	138 switch (t.kind)
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	139 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	140 case TOK.DATE:
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	141 time_str = Time.month_day(time_str) ~ ' ' ~ Time.year(time_str); break;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	142 case TOK.TIME:
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	143 time_str = Time.time(time_str); break;
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	144 case TOK.TIMESTAMP:
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	145 break; // time_str is the timestamp.
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	146 default: assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	147 }
776 580d4ca9f1ff Added new module dil.Time. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 769 diff changeset	148 time_str ~= '\0'; // Terminate with a zero.
343 95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	149 t.str = time_str;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	150 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	151 case TOK.VENDOR:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	152 t.str = VENDOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	153 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	154 case TOK.VERSION:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	155 t.uint_ = VERSION_MAJOR*1000 + VERSION_MINOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	156 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	157 default:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	158 assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	159 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	160 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token. aziz parents: 327 diff changeset	161
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	162 /// Sets a new file path.
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	163 void newFilePath(char[] newPath)
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	164 {
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	165 auto paths = new NewlineData.FilePaths;
755 90668b83ae5e Introduced new module dil.SourceText and class SourceText. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 742 diff changeset	166 paths.oriPath = this.srcText.filePath;
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	167 paths.setPath = newPath;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	168 this.filePaths = paths;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	169 }
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	170
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	171 private void setLineBegin(char* p)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	172 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	173 // Check that we can look behind one character.
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	174 assert((p-1) >= text.ptr && p < end);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	175 // Check that previous character is a newline.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	176 assert(isNewlineEnd(p - 1));
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	177 this.lineBegin = p;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	178 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	179
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	180 /// Scans the next token in the source text.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	181 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	182 /// Creates a new token if t.next is null and appends it to the list.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	183 private void scanNext(ref Token* t)
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	184 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	185 assert(t !is null);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	186 if (t.next)
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	187 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	188 t = t.next;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	189 // if (t.kind == TOK.Newline)
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	190 // this.newline = t;
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	191 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	192 else if (t != this.tail)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	193 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	194 Token* new_t = new Token;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	195 scan(*new_t);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	196 new_t.prev = t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	197 t.next = new_t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	198 t = new_t;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	199 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	200 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	201
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	202 /// Advance t one token forward.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	203 void peek(ref Token* t)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	204 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	205 scanNext(t);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	206 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	207
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	208 /// Advance to the next token in the source text.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	209 TOK nextToken()
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	210 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	211 scanNext(this.token);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	212 return this.token.kind;
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	213 }
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	214
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	215 /// Returns true if p points to the last character of a Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	216 bool isNewlineEnd(char* p)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	217 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	218 if (p == '\n' \|\| p == '\r')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	219 return true;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	220 if (p == LS[2] \|\| p == PS[2])
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	221 if ((p-2) >= text.ptr)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	222 if (p[-1] == LS[1] && p[-2] == LS[0])
434 cac9c8d637ad Fix: members related to error info weren't updated correctly. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 433 diff changeset	223 return true;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	224 return false;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	225 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	226
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	227 /// The main method which recognizes the characters that make up a token.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	228 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	229 /// Complicated tokens are scanned in separate methods.
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	230 public void scan(ref Token t)
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	231 in
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	232 {
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	233 assert(text.ptr <= p && p < end);
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	234 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	235 out
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	236 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	237 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	238 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	239 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	240 body
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	241 {
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	242 // Scan whitespace.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	243 if (isspace(*p))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	244 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	245 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	246 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	247 {}
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	248 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	249
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	250 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	251 uint c = *p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	252 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	253 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	254 // Newline.
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	255 switch (*p)
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	256 {
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	257 case '\r':
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	258 if (p[1] == '\n')
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	259 ++p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	260 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	261 assert(isNewlineEnd(p));
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	262 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	263 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	264 setLineBegin(p);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	265 // this.newline = &t;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	266 t.kind = TOK.Newline;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	267 t.setWhitespaceFlag();
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	268 t.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	269 t.newline.oriLineNum = lineNum;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	270 t.newline.setLineNum = lineNum_hline;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	271 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	272 return;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	273 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	274 if (isUnicodeNewline(p))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	275 {
412 fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	276 ++p; ++p;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 411 diff changeset	277 goto case '\n';
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	278 }
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	279 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	280 // Identifier or string literal.
13 e5211758b63c - Added isidbeg() function. aziz parents: 12 diff changeset	281 if (isidbeg(c))
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	282 {
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	283 if (c == 'r' && p[1] == '"' && ++p)
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	284 return scanRawStringLiteral(t);
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	285 if (c == 'x' && p[1] == '"')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	286 return scanHexStringLiteral(t);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	287 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	288 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	289 if (c == 'q' && p[1] == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	290 return scanDelimitedStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	291 if (c == 'q' && p[1] == '{')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	292 return scanTokenStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	293 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	294 // Scan identifier.
12 0989206cf73c - Added code to decode Unicode characters in identifiers. aziz parents: 11 diff changeset	295 Lidentifier:
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	296 do
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	297 { c = *++p; }
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	298 while (isident(c) \|\| !isascii(c) && isUnicodeAlpha())
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	299
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	300 t.end = p;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	301
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 499 diff changeset	302 auto id = IdTable.lookup(t.srcText);
769 5e3ef1b2011c Added and improved documentation. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 764 diff changeset	303 t.kind = id.kind;
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 499 diff changeset	304 t.ident = id;
28 3a9daccf7d96 - Added table for identifiers to Lexer. aziz parents: 27 diff changeset	305
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	306 if (t.kind == TOK.Identifier \|\| t.isKeyword)
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	307 return;
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	308 else if (t.isSpecialToken)
0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	309 finalizeSpecialToken(t);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	310 else if (t.kind == TOK.EOF)
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	311 {
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	312 tail = &t;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	313 assert(t.srcText == "__EOF__");
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	314 }
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	315 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	316 assert(0, "unexpected token type: " ~ Token.toString(t.kind));
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	317 return;
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	318 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	319
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	320 if (isdigit(c))
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	321 return scanNumber(t);
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	322
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	323 if (c == '/')
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	324 {
8 d4ba94a5a282 - Parsing /* / comments now. aziz* parents: 7 diff changeset	325 c = *++p;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	326 switch(c)
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	327 {
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	328 case '=':
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	329 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	330 t.kind = TOK.DivAssign;
14 cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	331 t.end = p;
cdf788d8bdaf - Parsing /= now. aziz parents: 13 diff changeset	332 return;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	333 case '+':
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	334 return scanNestedComment(t);
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	335 case '*':
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	336 return scanBlockComment(t);
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	337 case '/':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	338 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	339 isascii(*p) \|\| decodeUTF8();
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	340 t.kind = TOK.Comment;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	341 t.setWhitespaceFlag();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	342 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	343 return;
32 d7011daa4740 - Added missing commas after the items in the messages table. aziz parents: 31 diff changeset	344 default:
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	345 t.kind = TOK.Div;
10 3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	346 t.end = p;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	347 return;
3ee65d6e39c9 - Parsing // comments now. aziz parents: 9 diff changeset	348 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	349 }
79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	350
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	351 switch (c)
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	352 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	353 case '\'':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	354 return scanCharacterLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	355 case '`':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	356 return scanRawStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	357 case '"':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	358 return scanNormalStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	359 case '\\':
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	360 char[] buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	361 do
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	362 {
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	363 bool isBinary;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	364 c = scanEscapeSequence(isBinary);
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	365 if (isascii(c) \|\| isBinary)
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	366 buffer ~= c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	367 else
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	368 encodeUTF8(buffer, c);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	369 } while (*p == '\\')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	370 buffer ~= 0;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	371 t.kind = TOK.String;
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	372 t.str = buffer;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	373 t.end = p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	374 return;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	375 case '>': /* > >= >> >>= >>> >>>= */
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	376 c = *++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	377 switch (c)
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	378 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	379 case '=':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	380 t.kind = TOK.GreaterEqual;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	381 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	382 case '>':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	383 if (p[1] == '>')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	384 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	385 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	386 if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	387 { ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	388 t.kind = TOK.URShiftAssign;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	389 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	390 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	391 t.kind = TOK.URShift;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	392 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	393 else if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	394 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	395 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	396 t.kind = TOK.RShiftAssign;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	397 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	398 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	399 t.kind = TOK.RShift;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	400 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	401 default:
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	402 t.kind = TOK.Greater;
38 640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	403 goto Lcommon2;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	404 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens. aziz parents: 37 diff changeset	405 assert(0);
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	406 case '<': /* < <= <> <>= << <<= */
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	407 c = *++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	408 switch (c)
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	409 {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	410 case '=':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	411 t.kind = TOK.LessEqual;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	412 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	413 case '<':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	414 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	415 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	416 t.kind = TOK.LShiftAssign;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	417 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	418 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	419 t.kind = TOK.LShift;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	420 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	421 case '>':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	422 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	423 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	424 t.kind = TOK.LorEorG;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	425 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	426 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	427 t.kind = TOK.LorG;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	428 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	429 default:
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	430 t.kind = TOK.Less;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	431 goto Lcommon2;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	432 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	433 assert(0);
37 7f3bcb97d017 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 36 diff changeset	434 case '!': /* ! !< !> !<= !>= !<> !<>= */
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	435 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	436 switch (c)
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	437 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	438 case '<':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	439 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	440 if (c == '>')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	441 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	442 if (p[1] == '=') {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	443 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	444 t.kind = TOK.Unordered;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	445 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	446 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	447 t.kind = TOK.UorE;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	448 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	449 else if (c == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	450 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	451 t.kind = TOK.UorG;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	452 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	453 else {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	454 t.kind = TOK.UorGorE;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	455 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	456 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	457 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	458 case '>':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	459 if (p[1] == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	460 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	461 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	462 t.kind = TOK.UorL;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	463 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	464 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	465 t.kind = TOK.UorLorE;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	466 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	467 case '=':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	468 t.kind = TOK.NotEqual;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	469 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	470 default:
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	471 t.kind = TOK.Not;
35 c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	472 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	473 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens. aziz parents: 34 diff changeset	474 assert(0);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	475 case '.': /* . .[0-9] .. ... */
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	476 if (p[1] == '.')
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	477 {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	478 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	479 if (p[1] == '.') {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	480 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	481 t.kind = TOK.Ellipses;
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	482 }
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	483 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	484 t.kind = TOK.Slice;
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	485 }
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	486 else if (isdigit(p[1]))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	487 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	488 return scanReal(t);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	489 }
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	490 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	491 t.kind = TOK.Dot;
22 b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens. aziz parents: 21 diff changeset	492 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	493 case '\|': /* \| \|\| \|= */
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	494 c = *++p;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	495 if (c == '=')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	496 t.kind = TOK.OrAssign;
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	497 else if (c == '\|')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	498 t.kind = TOK.OrLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	499 else {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	500 t.kind = TOK.OrBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	501 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	502 }
23 1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens. aziz parents: 22 diff changeset	503 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	504 case '&': /* & && &= */
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	505 c = *++p;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	506 if (c == '=')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	507 t.kind = TOK.AndAssign;
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	508 else if (c == '&')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	509 t.kind = TOK.AndLogical;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	510 else {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	511 t.kind = TOK.AndBinary;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	512 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	513 }
24 903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens. aziz parents: 23 diff changeset	514 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	515 case '+': /* + ++ += */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	516 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	517 if (c == '=')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	518 t.kind = TOK.PlusAssign;
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	519 else if (c == '+')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	520 t.kind = TOK.PlusPlus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	521 else {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	522 t.kind = TOK.Plus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	523 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	524 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	525 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	526 case '-': /* - -- -= */
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	527 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	528 if (c == '=')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	529 t.kind = TOK.MinusAssign;
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	530 else if (c == '-')
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	531 t.kind = TOK.MinusMinus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	532 else {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	533 t.kind = TOK.Minus;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	534 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	535 }
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	536 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	537 case '=': /* = == */
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	538 if (p[1] == '=') {
c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	539 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	540 t.kind = TOK.Equal;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	541 }
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	542 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	543 t.kind = TOK.Assign;
25 9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens. aziz parents: 24 diff changeset	544 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	545 case '~': /* ~ ~= */
27 43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	546 if (p[1] == '=') {
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	547 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	548 t.kind = TOK.CatAssign;
27 43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	549 }
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	550 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	551 t.kind = TOK.Tilde;
27 43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens. aziz parents: 26 diff changeset	552 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	553 case '': / * = /
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	554 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	555 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	556 t.kind = TOK.MulAssign;
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	557 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	558 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	559 t.kind = TOK.Mul;
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	560 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	561 case '^': /* ^ ^= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	562 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	563 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	564 t.kind = TOK.XorAssign;
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	565 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	566 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	567 t.kind = TOK.Xor;
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	568 goto Lcommon;
36 3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens. aziz parents: 35 diff changeset	569 case '%': /* % %= */
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	570 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	571 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	572 t.kind = TOK.ModAssign;
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	573 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	574 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	575 t.kind = TOK.Mod;
29 ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	576 goto Lcommon;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens. aziz parents: 28 diff changeset	577 // Single character tokens:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	578 case '(':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	579 t.kind = TOK.LParen;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	580 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	581 case ')':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	582 t.kind = TOK.RParen;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	583 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	584 case '[':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	585 t.kind = TOK.LBracket;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	586 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	587 case ']':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	588 t.kind = TOK.RBracket;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	589 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	590 case '{':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	591 t.kind = TOK.LBrace;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	592 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	593 case '}':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	594 t.kind = TOK.RBrace;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	595 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	596 case ':':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	597 t.kind = TOK.Colon;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	598 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	599 case ';':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	600 t.kind = TOK.Semicolon;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	601 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	602 case '?':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	603 t.kind = TOK.Question;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	604 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	605 case ',':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	606 t.kind = TOK.Comma;
21 c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	607 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar. aziz parents: 20 diff changeset	608 case '$':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	609 t.kind = TOK.Dollar;
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	610 Lcommon:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	611 ++p;
26 c3d7373db241 - Added code for parsing Assign and Equal tokens. aziz parents: 25 diff changeset	612 Lcommon2:
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	613 t.end = p;
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	614 return;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	615 case '#':
360 b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence(). aziz parents: 350 diff changeset	616 return scanSpecialTokenSequence(t);
20 d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	617 default:
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	618 }
d6adfbd7c513 - Added code for parsing braces. aziz parents: 19 diff changeset	619
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	620 // Check for EOF
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	621 if (isEOF(c))
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	622 {
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	623 assert(isEOF(p), ""~p);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	624 t.kind = TOK.EOF;
411 cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	625 t.end = p;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	626 tail = &t;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	627 assert(t.start == t.end);
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	628 return;
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	629 }
cca83c0c00fd Added __EOF__ token. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 410 diff changeset	630
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	631 if (!isascii(c))
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	632 {
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	633 c = decodeUTF8();
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	634 if (isUniAlpha(c))
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	635 goto Lidentifier;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	636 }
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	637
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	638 error(t.start, MID.IllegalCharacter, cast(dchar)c);
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	639
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	640 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	641 t.kind = TOK.Illegal;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	642 t.setWhitespaceFlag();
390 4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	643 t.dchar_ = c;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	644 t.end = p;
4d36eea1bbc9 Refactored Lexer.scan(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 389 diff changeset	645 return;
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	646 }
3 4bbce78bfb1e - Added TOK enum. aziz parents: 2 diff changeset	647 }
4 92df59b1ec4a - Started implementation of scan(). aziz parents: 3 diff changeset	648
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	649 /// Converts a string literal to an integer.
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	650 template toUint(char[] T)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	651 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	652 static assert(0 < T.length && T.length <= 4);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	653 static if (T.length == 1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	654 const uint toUint = T[0];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	655 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	656 const uint toUint = (T[0] << ((T.length-1)*8)) \| toUint!(T[1..$]);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	657 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	658 static assert(toUint!("\xAA\xBB\xCC\xDD") == 0xAABBCCDD);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	659
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	660 /// Constructs case statements. E.g.:
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	661 /// ---
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	662 //// // case_!("<", "Less", "Lcommon") ->
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	663 /// case 60u:
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	664 /// t.kind = TOK.Less;
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	665 /// goto Lcommon;
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	666 /// ---
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	667 /// Note:Can't use this yet due to a $(DMDBUG 1534, bug) in DMD.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	668 template case_(char[] str, char[] kind, char[] label)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	669 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	670 const char[] case_ =
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	671 `case `~toUint!(str).stringof~`:`
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	672 `t.kind = TOK.`~kind~`;`
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	673 `goto `~label~`;`;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	674 }
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	675 //pragma(msg, case_!("<", "Less", "Lcommon"));
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	676
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	677 template case_L4(char[] str, TOK kind)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	678 {
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	679 const char[] case_L4 = case_!(str, kind, "Lcommon_4");
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	680 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	681
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	682 template case_L3(char[] str, TOK kind)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	683 {
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	684 const char[] case_L3 = case_!(str, kind, "Lcommon_3");
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	685 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	686
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	687 template case_L2(char[] str, TOK kind)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	688 {
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	689 const char[] case_L2 = case_!(str, kind, "Lcommon_2");
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	690 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	691
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	692 template case_L1(char[] str, TOK kind)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	693 {
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	694 const char[] case_L3 = case_!(str, kind, "Lcommon");
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	695 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	696
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	697 /// An alternative scan method.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	698 /// Profiling shows it's a bit slower.
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	699 public void scan_(ref Token t)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	700 in
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	701 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	702 assert(text.ptr <= p && p < end);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	703 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	704 out
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	705 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	706 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	707 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	708 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	709 body
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	710 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	711 // Scan whitespace.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	712 if (isspace(*p))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	713 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	714 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	715 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	716 {}
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	717 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	718
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	719 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	720 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	721 // Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	722 switch (*p)
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	723 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	724 case '\r':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	725 if (p[1] == '\n')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	726 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	727 case '\n':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	728 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	729 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	730 ++lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	731 setLineBegin(p);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	732 // this.newline = &t;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	733 t.kind = TOK.Newline;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	734 t.setWhitespaceFlag();
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	735 t.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	736 t.newline.oriLineNum = lineNum;
2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	737 t.newline.setLineNum = lineNum_hline;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	738 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	739 return;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	740 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	741 if (isUnicodeNewline(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	742 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	743 ++p; ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	744 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	745 }
418 7354f15cd5e9 Applied some fixes to the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 416 diff changeset	746 }
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	747
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	748 uint c = *p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	749 assert(end - p != 0);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	750 switch (end - p)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	751 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	752 case 1:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	753 goto L1character;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	754 case 2:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	755 c <<= 8; c \|= p[1];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	756 goto L2characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	757 case 3:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	758 c <<= 8; c \|= p[1]; c <<= 8; c \|= p[2];
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	759 goto L3characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	760 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	761 version(BigEndian)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	762 c = cast(uint)p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	763 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	764 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	765 c <<= 8; c \|= p[1]; c <<= 8; c \|= p[2]; c <<= 8; c \|= p[3];
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	766 /+
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	767 c = cast(uint)p;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	768 asm
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	769 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	770 mov EDX, c;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	771 bswap EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	772 mov c, EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	773 }
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	774 +/
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	775 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	776 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	777
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	778 // 4 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	779 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	780 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	781 case toUint!(">>>="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	782 t.kind = TOK.RShiftAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	783 goto Lcommon_4;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	784 case toUint!("!<>="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	785 t.kind = TOK.Unordered;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	786 Lcommon_4:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	787 p += 4;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	788 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	789 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	790 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	791 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	792
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	793 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	794 L3characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	795 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	796 // 3 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	797 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	798 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	799 case toUint!(">>="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	800 t.kind = TOK.RShiftAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	801 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	802 case toUint!(">>>"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	803 t.kind = TOK.URShift;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	804 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	805 case toUint!("<>="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	806 t.kind = TOK.LorEorG;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	807 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	808 case toUint!("<<="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	809 t.kind = TOK.LShiftAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	810 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	811 case toUint!("!<="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	812 t.kind = TOK.UorG;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	813 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	814 case toUint!("!>="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	815 t.kind = TOK.UorL;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	816 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	817 case toUint!("!<>"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	818 t.kind = TOK.UorE;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	819 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	820 case toUint!("..."):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	821 t.kind = TOK.Ellipses;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	822 Lcommon_3:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	823 p += 3;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	824 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	825 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	826 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	827 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	828
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	829 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	830 L2characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	831 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	832 // 2 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	833 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	834 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	835 case toUint!("/+"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	836 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	837 return scanNestedComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	838 case toUint!("/*"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	839 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	840 return scanBlockComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	841 case toUint!("//"):
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	842 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	843 assert(*p == '/');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	844 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	845 isascii(*p) \|\| decodeUTF8();
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	846 t.kind = TOK.Comment;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	847 t.setWhitespaceFlag();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	848 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	849 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	850 case toUint!(">="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	851 t.kind = TOK.GreaterEqual;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	852 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	853 case toUint!(">>"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	854 t.kind = TOK.RShift;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	855 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	856 case toUint!("<<"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	857 t.kind = TOK.LShift;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	858 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	859 case toUint!("<="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	860 t.kind = TOK.LessEqual;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	861 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	862 case toUint!("<>"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	863 t.kind = TOK.LorG;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	864 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	865 case toUint!("!<"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	866 t.kind = TOK.UorGorE;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	867 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	868 case toUint!("!>"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	869 t.kind = TOK.UorLorE;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	870 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	871 case toUint!("!="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	872 t.kind = TOK.NotEqual;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	873 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	874 case toUint!(".."):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	875 t.kind = TOK.Slice;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	876 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	877 case toUint!("&&"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	878 t.kind = TOK.AndLogical;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	879 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	880 case toUint!("&="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	881 t.kind = TOK.AndAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	882 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	883 case toUint!("\|\|"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	884 t.kind = TOK.OrLogical;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	885 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	886 case toUint!("\|="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	887 t.kind = TOK.OrAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	888 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	889 case toUint!("++"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	890 t.kind = TOK.PlusPlus;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	891 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	892 case toUint!("+="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	893 t.kind = TOK.PlusAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	894 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	895 case toUint!("--"):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	896 t.kind = TOK.MinusMinus;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	897 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	898 case toUint!("-="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	899 t.kind = TOK.MinusAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	900 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	901 case toUint!("=="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	902 t.kind = TOK.Equal;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	903 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	904 case toUint!("~="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	905 t.kind = TOK.CatAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	906 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	907 case toUint!("*="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	908 t.kind = TOK.MulAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	909 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	910 case toUint!("/="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	911 t.kind = TOK.DivAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	912 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	913 case toUint!("^="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	914 t.kind = TOK.XorAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	915 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	916 case toUint!("%="):
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	917 t.kind = TOK.ModAssign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	918 Lcommon_2:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	919 p += 2;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	920 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	921 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	922 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	923 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	924
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	925 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	926 L1character:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	927 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	928 assert(p == c, Format("p={0},c={1}", p, cast(dchar)c));
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	929 // 1 character tokens.
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	930 // TODO: consider storing the token type in ptable.
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	931 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	932 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	933 case '\'':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	934 return scanCharacterLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	935 case '`':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	936 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	937 case '"':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	938 return scanNormalStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	939 case '\\':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	940 char[] buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	941 do
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	942 {
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	943 bool isBinary;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	944 c = scanEscapeSequence(isBinary);
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	945 if (isascii(c) \|\| isBinary)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	946 buffer ~= c;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	947 else
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	948 encodeUTF8(buffer, c);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	949 } while (*p == '\\')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	950 buffer ~= 0;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	951 t.kind = TOK.String;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	952 t.str = buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	953 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	954 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	955 case '<':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	956 t.kind = TOK.Greater;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	957 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	958 case '>':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	959 t.kind = TOK.Less;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	960 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	961 case '^':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	962 t.kind = TOK.Xor;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	963 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	964 case '!':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	965 t.kind = TOK.Not;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	966 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	967 case '.':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	968 if (isdigit(p[1]))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	969 return scanReal(t);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	970 t.kind = TOK.Dot;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	971 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	972 case '&':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	973 t.kind = TOK.AndBinary;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	974 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	975 case '\|':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	976 t.kind = TOK.OrBinary;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	977 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	978 case '+':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	979 t.kind = TOK.Plus;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	980 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	981 case '-':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	982 t.kind = TOK.Minus;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	983 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	984 case '=':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	985 t.kind = TOK.Assign;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	986 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	987 case '~':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	988 t.kind = TOK.Tilde;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	989 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	990 case '*':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	991 t.kind = TOK.Mul;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	992 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	993 case '/':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	994 t.kind = TOK.Div;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	995 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	996 case '%':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	997 t.kind = TOK.Mod;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	998 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	999 case '(':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1000 t.kind = TOK.LParen;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1001 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1002 case ')':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1003 t.kind = TOK.RParen;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1004 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1005 case '[':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1006 t.kind = TOK.LBracket;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1007 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1008 case ']':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1009 t.kind = TOK.RBracket;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1010 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1011 case '{':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1012 t.kind = TOK.LBrace;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1013 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1014 case '}':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1015 t.kind = TOK.RBrace;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1016 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1017 case ':':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1018 t.kind = TOK.Colon;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1019 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1020 case ';':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1021 t.kind = TOK.Semicolon;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1022 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1023 case '?':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1024 t.kind = TOK.Question;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1025 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1026 case ',':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1027 t.kind = TOK.Comma;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1028 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1029 case '$':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1030 t.kind = TOK.Dollar;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1031 Lcommon:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1032 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1033 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1034 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1035 case '#':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1036 return scanSpecialTokenSequence(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1037 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1038 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1039
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1040 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1041 assert(*p == c);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1042
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1043 // TODO: consider moving isidbeg() and isdigit() up.
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1044 if (isidbeg(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1045 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1046 if (c == 'r' && p[1] == '"' && ++p)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1047 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1048 if (c == 'x' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1049 return scanHexStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1050 version(D2)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1051 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1052 if (c == 'q' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1053 return scanDelimitedStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1054 if (c == 'q' && p[1] == '{')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1055 return scanTokenStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1056 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1057 // Scan identifier.
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1058 Lidentifier:
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1059 do
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1060 { c = *++p; }
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	1061 while (isident(c) \|\| !isascii(c) && isUnicodeAlpha())
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1062
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1063 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1064
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 499 diff changeset	1065 auto id = IdTable.lookup(t.srcText);
769 5e3ef1b2011c Added and improved documentation. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 764 diff changeset	1066 t.kind = id.kind;
502 4e14cd1b24da Refactored code and added modules related to tabulated Identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 499 diff changeset	1067 t.ident = id;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1068
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1069 if (t.kind == TOK.Identifier \|\| t.isKeyword)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1070 return;
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1071 else if (t.isSpecialToken)
0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1072 finalizeSpecialToken(t);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1073 else if (t.kind == TOK.EOF)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1074 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1075 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1076 assert(t.srcText == "__EOF__");
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1077 }
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1078 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1079 assert(0, "unexpected token type: " ~ Token.toString(t.kind));
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1080 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1081 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1082
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1083 if (isdigit(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1084 return scanNumber(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1085
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1086 // Check for EOF
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1087 if (isEOF(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1088 {
497 0ffcc4ff82f3 Refactored a few things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 496 diff changeset	1089 assert(isEOF(p), p~"");
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1090 t.kind = TOK.EOF;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1091 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1092 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1093 assert(t.start == t.end);
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1094 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1095 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1096
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1097 if (!isascii(c))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1098 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1099 c = decodeUTF8();
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1100 if (isUniAlpha(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1101 goto Lidentifier;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1102 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1103
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1104 error(t.start, MID.IllegalCharacter, cast(dchar)c);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1105
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1106 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1107 t.kind = TOK.Illegal;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	1108 t.setWhitespaceFlag();
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1109 t.dchar_ = c;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1110 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1111 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1112 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1113
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1114 /// Scans a block comment.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1115 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1116 /// BlockComment := "/" AnyChar "*/"
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1117 void scanBlockComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1118 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1119 assert(p[-1] == '/' && p == '');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1120 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1121 auto tokenLineBegin = lineBegin;
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1122 Loop:
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1123 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1124 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1125 switch (*++p)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1126 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1127 case '*':
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1128 if (p[1] != '/')
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1129 continue;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1130 p += 2;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1131 break Loop;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1132 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1133 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1134 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1135 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1136 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1137 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1138 setLineBegin(p+1);
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1139 break;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1140 default:
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1141 if (!isascii(*p))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1142 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1143 if (isUnicodeNewlineChar(decodeUTF8()))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1144 goto case '\n';
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1145 }
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1146 else if (isEOF(*p))
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1147 {
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1148 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedBlockComment);
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1149 break Loop;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1150 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1151 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1152 }
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1153 t.kind = TOK.Comment;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	1154 t.setWhitespaceFlag();
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1155 t.end = p;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1156 return;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1157 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1158
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1159 /// Scans a nested comment.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1160 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1161 /// NestedComment := "/+" (AnyChar* \| NestedComment) "+/"
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1162 void scanNestedComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1163 {
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1164 assert(p[-1] == '/' && *p == '+');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1165 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1166 auto tokenLineBegin = lineBegin;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1167 uint level = 1;
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1168 Loop:
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1169 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1170 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1171 switch (*++p)
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1172 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1173 case '/':
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1174 if (p[1] == '+')
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1175 ++p, ++level;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1176 continue;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1177 case '+':
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1178 if (p[1] != '/')
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1179 continue;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1180 ++p;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1181 if (--level != 0)
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1182 continue;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1183 ++p;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1184 break Loop;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1185 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1186 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1187 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1188 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1189 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1190 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1191 setLineBegin(p+1);
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1192 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1193 default:
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1194 if (!isascii(*p))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1195 {
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1196 if (isUnicodeNewlineChar(decodeUTF8()))
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1197 goto case '\n';
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1198 }
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1199 else if (isEOF(*p))
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1200 {
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1201 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedNestedComment);
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1202 break Loop;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1203 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1204 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1205 }
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1206 t.kind = TOK.Comment;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	1207 t.setWhitespaceFlag();
498 49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1208 t.end = p;
49c201b5c465 Refactored scanners for block and nested comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 497 diff changeset	1209 return;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1210 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	1211
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1212 /// Scans the postfix character of a string literal.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1213 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1214 /// PostfixChar := "c" \| "w" \| "d"
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1215 char scanPostfix()
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1216 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1217 assert(p[-1] == '"' \|\| p[-1] == '`' \|\|
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1218 { version(D2) return p[-1] == '}';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1219 else return 0; }()
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1220 );
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1221 switch (*p)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1222 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1223 case 'c':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1224 case 'w':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1225 case 'd':
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1226 return *p++;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1227 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1228 return 0;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1229 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1230 assert(0);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1231 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1232
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1233 /// Scans a normal string literal.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1234 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1235 /// NormalStringLiteral := "\"" Char* "\""
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1236 void scanNormalStringLiteral(ref Token t)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1237 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1238 assert(*p == '"');
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1239 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1240 auto tokenLineBegin = lineBegin;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1241 t.kind = TOK.String;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1242 char[] buffer;
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1243 uint c;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1244 while (1)
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1245 {
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1246 c = *++p;
f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1247 switch (c)
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1248 {
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1249 case '"':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1250 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1251 t.pf = scanPostfix();
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1252 Lreturn:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1253 t.str = buffer ~ '\0';
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1254 t.end = p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1255 return;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1256 case '\\':
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1257 bool isBinary;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1258 c = scanEscapeSequence(isBinary);
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1259 --p;
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1260 if (isascii(c) \|\| isBinary)
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1261 buffer ~= c;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1262 else
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1263 encodeUTF8(buffer, c);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1264 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1265 case '\r':
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1266 if (p[1] == '\n')
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1267 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1268 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1269 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1270 c = '\n'; // Convert Newline to \n.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1271 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1272 setLineBegin(p+1);
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1273 break;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1274 case 0, _Z_:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1275 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedString);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1276 goto Lreturn;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1277 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1278 if (!isascii(c))
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1279 {
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1280 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1281 if (isUnicodeNewlineChar(c))
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1282 goto case '\n';
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1283 encodeUTF8(buffer, c);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1284 continue;
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1285 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1286 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1287 assert(isascii(c));
384 f7ce725e79c3 Refactored scanNormalStringLiteral(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 383 diff changeset	1288 buffer ~= c;
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1289 }
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	1290 assert(0);
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1291 }
8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	1292
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1293 /// Scans a character literal.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1294 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1295 /// CharLiteral := "'" Char "'"
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1296 void scanCharacterLiteral(ref Token t)
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1297 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1298 assert(*p == '\'');
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1299 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1300 t.kind = TOK.CharLiteral;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1301 switch (*p)
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1302 {
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1303 case '\\':
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1304 bool notused;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1305 t.dchar_ = scanEscapeSequence(notused);
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1306 break;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1307 case '\'':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1308 error(t.start, MID.EmptyCharacterLiteral);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1309 break;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1310 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1311 if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1312 break;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1313 uint c = *p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1314 if (!isascii(c))
49 7f0fa15dcffc - Renamed function. aziz parents: 48 diff changeset	1315 c = decodeUTF8();
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1316 t.dchar_ = c;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1317 ++p;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1318 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1319
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1320 if (*p == '\'')
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1321 ++p;
e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1322 else
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1323 error(t.start, MID.UnterminatedCharacterLiteral);
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1324 t.end = p;
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1325 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	1326
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1327 /// Scans a raw string literal.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1328 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1329 /// RawStringLiteral := "r\"" AnyChar* "\"" \| "`" AnyChar* "`"
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1330 void scanRawStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1331 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1332 assert(p == '`' \|\| p == '"' && p[-1] == 'r');
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1333 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1334 auto tokenLineBegin = lineBegin;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1335 t.kind = TOK.String;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1336 uint delim = *p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1337 char[] buffer;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1338 uint c;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1339 while (1)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1340 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1341 c = *++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1342 switch (c)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1343 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1344 case '\r':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1345 if (p[1] == '\n')
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1346 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1347 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1348 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1349 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1350 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1351 setLineBegin(p+1);
52 f65a83c27638 - Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well. aziz parents: 51 diff changeset	1352 break;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1353 case '`':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1354 case '"':
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1355 if (c == delim)
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1356 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1357 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1358 t.pf = scanPostfix();
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1359 Lreturn:
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1360 t.str = buffer ~ '\0';
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1361 t.end = p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1362 return;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1363 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1364 break;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1365 case 0, _Z_:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1366 error(tokenLineNum, tokenLineBegin, t.start,
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1367 delim == 'r' ? MID.UnterminatedRawString : MID.UnterminatedBackQuoteString);
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1368 goto Lreturn;
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1369 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1370 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1371 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1372 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1373 if (isUnicodeNewlineChar(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1374 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1375 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1376 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1377 }
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1378 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1379 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1380 buffer ~= c;
33 cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1381 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings. aziz parents: 32 diff changeset	1382 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1383 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1384
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1385 /// Scans a hexadecimal string literal.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1386 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1387 /// HexStringLiteral := "x\"" (HexChar HexChar)* "\""
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1388 void scanHexStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1389 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1390 assert(p[0] == 'x' && p[1] == '"');
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1391 t.kind = TOK.String;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1392
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1393 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1394 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1395
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1396 uint c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1397 ubyte[] buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1398 ubyte h; // hex number
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1399 uint n; // number of hex digits
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1400
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1401 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1402 assert(*p == '"');
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1403 while (1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1404 {
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1405 c = *++p;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1406 switch (c)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1407 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1408 case '"':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1409 if (n & 1)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1410 error(tokenLineNum, tokenLineBegin, t.start, MID.OddNumberOfDigitsInHexString);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1411 ++p;
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1412 t.pf = scanPostfix();
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1413 Lreturn:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1414 t.str = cast(string) (buffer ~= 0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1415 t.end = p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1416 return;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1417 case '\r':
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1418 if (p[1] == '\n')
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1419 ++p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1420 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1421 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1422 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1423 setLineBegin(p+1);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1424 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1425 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1426 if (ishexad(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1427 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1428 if (c <= '9')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1429 c -= '0';
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1430 else if (c <= 'F')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1431 c -= 'A' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1432 else
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1433 c -= 'a' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1434
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1435 if (n & 1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1436 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1437 h <<= 4;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1438 h \|= c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1439 buffer ~= h;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1440 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1441 else
112 004d98df65af - Implemented parseInterfaceDeclaration(). aziz parents: 105 diff changeset	1442 h = cast(ubyte)c;
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1443 ++n;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1444 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1445 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1446 else if (isspace(c))
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1447 continue; // Skip spaces.
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 494 diff changeset	1448 else if (isEOF(c))
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1449 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1450 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedHexString);
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1451 t.pf = 0;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1452 goto Lreturn;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1453 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1454 else
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1455 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1456 auto errorAt = p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1457 if (!isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1458 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1459 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1460 if (isUnicodeNewlineChar(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1461 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1462 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1463 error(errorAt, MID.NonHexCharInHexString, cast(dchar)c);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1464 }
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1465 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1466 }
53 1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom. aziz parents: 52 diff changeset	1467 assert(0);
31 94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1468 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix. aziz parents: 30 diff changeset	1469
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1470 version(DDoc)
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1471 {
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1472 /// Scans a delimited string literal.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1473 void scanDelimitedStringLiteral(ref Token t);
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1474 /// Scans a token string literal.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1475 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1476 /// TokenStringLiteral := "q{" Token* "}"
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1477 void scanTokenStringLiteral(ref Token t);
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1478 }
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1479 else
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1480 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1481 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1482 void scanDelimitedStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1483 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1484 assert(p[0] == 'q' && p[1] == '"');
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1485 t.kind = TOK.String;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1486
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1487 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1488 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1489
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1490 char[] buffer;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1491 dchar opening_delim = 0, // 0 if no nested delimiter or '[', '(', '<', '{'
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1492 closing_delim; // Will be ']', ')', '>', '},
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1493 // the first character of an identifier or
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1494 // any other Unicode/ASCII character.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1495 char[] str_delim; // Identifier delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1496 uint level = 1; // Counter for nestable delimiters.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1497
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1498 ++p; ++p; // Skip q"
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1499 uint c = *p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1500 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1501 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1502 case '(':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1503 opening_delim = c;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1504 closing_delim = ')'; // c + 1
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1505 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1506 case '[', '<', '{':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1507 opening_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1508 closing_delim = c + 2; // Get to closing counterpart. Feature of ASCII table.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1509 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1510 default:
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1511 dchar scanNewline()
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1512 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1513 switch (*p)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1514 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1515 case '\r':
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1516 if (p[1] == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1517 ++p;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1518 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1519 assert(isNewlineEnd(p));
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1520 ++p;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1521 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1522 setLineBegin(p);
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1523 return '\n';
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1524 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1525 if (isUnicodeNewline(p))
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1526 {
419 89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1527 ++p; ++p;
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 418 diff changeset	1528 goto case '\n';
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1529 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1530 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1531 return 0;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1532 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1533 // Skip leading newlines:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1534 while (scanNewline() != 0)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1535 {}
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1536 assert(!isNewline(p));
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1537
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1538 char* begin = p;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1539 c = *p;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1540 closing_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1541 // TODO: Check for non-printable characters?
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1542 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1543 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1544 closing_delim = decodeUTF8();
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1545 if (!isUniAlpha(closing_delim))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1546 break; // Not an identifier.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1547 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1548 else if (!isidbeg(c))
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1549 break; // Not an identifier.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1550
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1551 // Parse Identifier + EndOfLine
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1552 do
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1553 { c = *++p; }
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	1554 while (isident(c) \|\| !isascii(c) && isUnicodeAlpha())
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1555 // Store identifier
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1556 str_delim = begin[0..p-begin];
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1557 // Scan newline
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1558 if (scanNewline() == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1559 --p; // Go back one because of "c = *++p;" in main loop.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1560 else
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1561 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1562 // TODO: error(p, MID.ExpectedNewlineAfterIdentDelim);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1563 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1564 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1565
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1566 bool checkStringDelim(char* p)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1567 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1568 assert(str_delim.length != 0);
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1569 if (buffer[$-1] == '\n' && // Last character copied to buffer must be '\n'.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1570 end-p >= str_delim.length && // Check remaining length.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1571 p[0..str_delim.length] == str_delim) // Compare.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1572 return true;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1573 return false;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1574 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1575
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1576 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1577 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1578 c = *++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1579 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1580 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1581 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1582 if (p[1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1583 ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1584 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1585 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1586 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1587 ++lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1588 setLineBegin(p+1);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1589 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1590 case 0, _Z_:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1591 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedDelimitedString);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1592 goto Lreturn3;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1593 default:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1594 if (!isascii(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1595 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1596 auto begin = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1597 c = decodeUTF8();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1598 if (isUnicodeNewlineChar(c))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1599 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1600 if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1601 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1602 if (str_delim.length)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1603 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1604 if (checkStringDelim(begin))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1605 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1606 p = begin + str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1607 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1608 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1609 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1610 else
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1611 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1612 assert(level == 1);
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1613 --level;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1614 goto Lreturn;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1615 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1616 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1617 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1618 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1619 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1620 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1621 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1622 if (c == opening_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1623 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1624 else if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1625 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1626 if (str_delim.length)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1627 {
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1628 if (checkStringDelim(p))
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1629 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1630 p += str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1631 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1632 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1633 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1634 else if (--level == 0)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1635 goto Lreturn;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1636 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1637 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1638 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1639 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1640 buffer ~= c;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1641 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1642 Lreturn: // Character delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1643 assert(c == closing_delim);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1644 assert(level == 0);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1645 ++p; // Skip closing delimiter.
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1646 Lreturn2: // String delimiter.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1647 if (*p == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1648 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1649 else
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1650 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1651 // TODO: error(p, MID.ExpectedDblQuoteAfterDelim, str_delim.length ? str_delim : closing_delim~"");
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1652 }
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1653
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1654 t.pf = scanPostfix();
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1655 Lreturn3: // Error.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1656 t.str = buffer ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1657 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1658 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1659
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1660 void scanTokenStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1661 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1662 assert(p[0] == 'q' && p[1] == '{');
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1663 t.kind = TOK.String;
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1664
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1665 auto tokenLineNum = lineNum;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1666 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1667
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1668 // A guard against changes to particular members:
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1669 // this.lineNum_hline and this.errorPath
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1670 ++inTokenString;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1671
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1672 uint lineNum = this.lineNum;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1673 uint level = 1;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1674
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1675 ++p; ++p; // Skip q{
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1676
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1677 auto prev_t = &t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1678 Token* token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1679 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1680 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1681 token = new Token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1682 scan(*token);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1683 // Save the tokens in a doubly linked list.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1684 // Could be useful for various tools.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1685 token.prev = prev_t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1686 prev_t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1687 prev_t = token;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1688 switch (token.kind)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1689 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1690 case TOK.LBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1691 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1692 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1693 case TOK.RBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1694 if (--level == 0)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1695 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1696 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1697 t.next = null;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1698 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1699 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1700 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1701 case TOK.EOF:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1702 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedTokenString);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1703 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1704 t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1705 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1706 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1707 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1708 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1709 break; // Exit loop.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1710 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1711
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1712 assert(token.kind == TOK.RBrace \|\| token.kind == TOK.EOF);
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1713 assert(token.kind == TOK.RBrace && t.next is null \|\|
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1714 token.kind == TOK.EOF && t.next !is null);
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1715
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1716 char[] buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1717 // token points to } or EOF
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	1718 if (token.kind == TOK.EOF)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1719 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1720 t.end = token.start;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1721 buffer = t.srcText[2..$].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1722 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1723 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1724 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1725 // Assign to buffer before scanPostfix().
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1726 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1727 buffer = t.srcText[2..$-1].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1728 t.pf = scanPostfix();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1729 t.end = p; // Assign again because of postfix.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1730 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1731 // Convert newlines to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1732 if (lineNum != this.lineNum)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1733 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1734 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1735 uint i, j;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1736 for (; i < buffer.length; ++i)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1737 switch (buffer[i])
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1738 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1739 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1740 if (buffer[i+1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1741 ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1742 case '\n':
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1743 assert(isNewlineEnd(buffer.ptr + i));
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1744 buffer[j++] = '\n'; // Convert Newline to '\n'.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1745 break;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1746 default:
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1747 if (isUnicodeNewline(buffer.ptr + i))
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1748 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1749 ++i; ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1750 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1751 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1752 buffer[j++] = buffer[i]; // Copy.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1753 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1754 buffer.length = j; // Adjust length.
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1755 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1756 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1757 t.str = buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1758
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1759 --inTokenString;
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1760 }
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1761 } // version(D2)
383 6a5fc22cae34 Implemented scanner for new string literals and applied some fixes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 382 diff changeset	1762
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1763 /// Scans an escape sequence.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1764 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1765 /// EscapeSequence := "\" (Octal{1,3} \| ("x" Hex{2}) \|
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1766 /// ("u" Hex{4}) \| ("U" Hex{8}) \|
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1767 /// "'" \| "\"" \| "\\" \| "?" \| "a" \|
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1768 /// "b" \| "f" \| "n" \| "r" \| "t" \| "v")
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1769 /// Params:
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1770 /// isBinary = set to true for octal and hexadecimal escapes.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1771 /// Returns: the escape value.
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1772 dchar scanEscapeSequence(ref bool isBinary)
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1773 out(result)
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	1774 { assert(isValidChar(result)); }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1775 body
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1776 {
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1777 assert(*p == '\\');
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1778
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1779 auto sequenceStart = p; // Used for error reporting.
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1780
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1781 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1782 uint c = char2ev(*p);
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1783 if (c)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1784 {
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1785 ++p;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1786 return c;
46 e0dbd4722f74 - Scanning character literals correctly now. aziz parents: 45 diff changeset	1787 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1788
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1789 uint digits = 2;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1790
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1791 switch (*p)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1792 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1793 case 'x':
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1794 isBinary = true;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1795 case_Unicode:
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1796 assert(c == 0);
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1797 assert(digits == 2 \|\| digits == 4 \|\| digits == 8);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1798 while (1)
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1799 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1800 ++p;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1801 if (ishexad(*p))
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1802 {
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1803 c *= 16;
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1804 if (*p <= '9')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1805 c += *p - '0';
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1806 else if (*p <= 'F')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1807 c += *p - 'A' + 10;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1808 else
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1809 c += *p - 'a' + 10;
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1810
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1811 if (--digits == 0)
82 fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user. aziz parents: 71 diff changeset	1812 {
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1813 ++p;
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	1814 if (isValidChar(c))
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1815 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1816
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1817 error(sequenceStart, MID.InvalidUnicodeEscapeSequence,
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1818 sequenceStart[0..p-sequenceStart]);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1819 break;
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1820 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1821 continue;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1822 }
432 3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 430 diff changeset	1823
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1824 error(sequenceStart, MID.InsufficientHexDigits,
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1825 sequenceStart[0..p-sequenceStart]);
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1826 break;
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	1827 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1828 break;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1829 case 'u':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1830 digits = 4;
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1831 goto case_Unicode;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1832 case 'U':
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1833 digits = 8;
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1834 goto case_Unicode;
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1835 default:
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1836 if (isoctal(*p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1837 {
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1838 isBinary = true;
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1839 assert(c == 0);
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1840 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1841 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1842 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1843 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1844 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1845 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1846 ++p;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1847 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1848 return c;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1849 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1850 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1851 ++p;
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	1852 return c & 0xFF; // Return valid escape value.
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1853 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1854 else if(*p == '&')
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1855 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1856 if (isalpha(*++p))
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1857 {
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1858 auto begin = p;
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1859 while (isalnum(*++p))
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1860 {}
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1861
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1862 if (*p == ';')
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1863 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1864 // Pass entity excluding '&' and ';'.
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1865 c = entity2Unicode(begin[0..p - begin]);
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1866 ++p; // Skip ;
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1867 if (c != 0xFFFF)
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1868 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1869 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1870 error(sequenceStart, MID.UndefinedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
272 0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1871 }
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters. aziz parents: 249 diff changeset	1872 else
432 3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 430 diff changeset	1873 error(sequenceStart, MID.UnterminatedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1874 }
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	1875 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1876 error(sequenceStart, MID.InvalidBeginHTMLEntity);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1877 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1878 else if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1879 error(sequenceStart, MID.UndefinedEscapeSequence,
496 5a607597dc22 Improved error recovery in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 494 diff changeset	1880 isEOF(*p) ? `\EOF` : `\NewLine`);
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1881 else
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1882 {
392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1883 char[] str = `\`;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1884 if (isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1885 str ~= *p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	1886 else
387 ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1887 encodeUTF8(str, decodeUTF8());
ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1888 ++p;
ad0cbd1c8881 Undefined escape sequences are passed to error() now. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 386 diff changeset	1889 // TODO: check for unprintable character?
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1890 error(sequenceStart, MID.UndefinedEscapeSequence, str);
386 392a0068fc61 Refactored code related to scanning escape sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 384 diff changeset	1891 }
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1892 }
433 063cd016c913 Fixed some issues in scanEscapeSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 432 diff changeset	1893 return REPLACEMENT_CHAR; // Error: return replacement character.
45 cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1894 }
cc84b9ca9d0a - Implemented escape sequence scanner. aziz parents: 44 diff changeset	1895
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1896 /// Scans a number literal.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1897 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1898 /// $(PRE
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1899 /// IntegerLiteral := (Dec\|Hex\|Bin\|Oct)Suffix?
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1900 /// Dec := (0\|[1-9][0-9_]*)
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1901 /// Hex := 0[xX][_][0-9a-zA-Z][0-9a-zA-Z_]
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1902 /// Bin := 0[bB][_][01][01_]
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1903 /// Oct := 0[0-7_]*
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1904 /// Suffix := (L[uU]?\|[uU]L?)
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1905 /// )
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	1906 /// Invalid: "0b_", "0x_", "._" etc.
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1907 void scanNumber(ref Token t)
c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	1908 {
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1909 ulong ulong_;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1910 bool overflow;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1911 bool isDecimal;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	1912 size_t digits;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1913
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1914 if (*p != '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1915 goto LscanInteger;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1916 ++p; // skip zero
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1917 // check for xX bB ...
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1918 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1919 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1920 case 'x','X':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1921 goto LscanHex;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1922 case 'b','B':
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1923 goto LscanBinary;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1924 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1925 if (p[1] == 'i')
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1926 goto LscanReal; // 0Li
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1927 break; // 0L
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1928 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1929 if (p[1] == '.')
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1930 break; // 0..
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1931 // 0.
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1932 case 'i','f','F', // Imaginary and float literal suffixes.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	1933 'e', 'E': // Float exponent.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1934 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1935 default:
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1936 if (*p == '_')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1937 goto LscanOctal; // 0_
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1938 else if (isdigit(*p))
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1939 {
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1940 if (p == '8' \|\| p == '9')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1941 goto Loctal_hasDecimalDigits; // 08 or 09
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1942 else
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1943 goto Loctal_enter_loop; // 0[0-7]
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1944 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1945 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1946
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1947 // Number 0
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1948 assert(p[-1] == '0');
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	1949 assert(p != '_' && !isdigit(p));
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1950 assert(ulong_ == 0);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1951 isDecimal = true;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1952 goto Lfinalize;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1953
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1954 LscanInteger:
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1955 assert(p != 0 && isdigit(p));
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1956 isDecimal = true;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1957 goto Lenter_loop_int;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1958 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1959 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1960 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1961 continue;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1962 if (!isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	1963 break;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	1964 Lenter_loop_int:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1965 if (ulong_ < ulong.max/10 \|\| (ulong_ == ulong.max/10 && *p <= '5'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1966 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1967 ulong_ *= 10;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1968 ulong_ += *p - '0';
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1969 continue;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1970 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1971 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1972 overflow = true;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1973 while (isdigit(*++p)) {}
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1974 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1975 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1976
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1977 // The number could be a float, so check overflow below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1978 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1979 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1980 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1981 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1982 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1983 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1984 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1985 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1986 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1987 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1988 goto LscanReal;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1989 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1990 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1991
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1992 if (overflow)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	1993 error(t.start, MID.OverflowDecimalNumber);
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	1994
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1995 assert((isdigit(p[-1]) \|\| p[-1] == '_') && !isdigit(p) && p != '_');
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1996 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1997
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	1998 LscanHex:
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	1999 assert(digits == 0);
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	2000 assert(p == 'x' \|\| p == 'X');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2001 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2002 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2003 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2004 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2005 if (!ishexad(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2006 break;
61 512cd2248dfc - Fix: issueing error on hexadecimal number overflow. aziz parents: 60 diff changeset	2007 ++digits;
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2008 ulong_ *= 16;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2009 if (*p <= '9')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2010 ulong_ += *p - '0';
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2011 else if (*p <= 'F')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2012 ulong_ += *p - 'A' + 10;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2013 else
c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2014 ulong_ += *p - 'a' + 10;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2015 }
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2016
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	2017 assert(ishexad(p[-1]) \|\| p[-1] == '_' \|\| p[-1] == 'x' \|\| p[-1] == 'X');
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2018 assert(!ishexad(p) && p != '_');
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2019
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2020 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2021 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2022 case '.':
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2023 if (p[1] == '.')
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2024 break;
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2025 case 'p', 'P':
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2026 return scanHexReal(t);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2027 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2028 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2029
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2030 if (digits == 0 \|\| digits > 16)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2031 error(t.start, digits == 0 ? MID.NoDigitsInHexNumber : MID.OverflowHexNumber);
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2032
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2033 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2034
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2035 LscanBinary:
57 c0f1c8be3a47 - Added code for converting hex characters to binary numbers. aziz parents: 56 diff changeset	2036 assert(digits == 0);
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	2037 assert(p == 'b' \|\| p == 'B');
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2038 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2039 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2040 if (*++p == '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2041 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2042 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2043 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2044 }
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2045 else if (*p == '1')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2046 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2047 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2048 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2049 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2050 }
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2051 else if (*p == '_')
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2052 continue;
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2053 else
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2054 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2055 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2056
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2057 if (digits == 0 \|\| digits > 64)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2058 error(t.start, digits == 0 ? MID.NoDigitsInBinNumber : MID.OverflowBinaryNumber);
59 3e594725899a - Issuing error when no digits were found in hex and binary numbers. aziz parents: 58 diff changeset	2059
410 4d9ee8e60712 Added destructors for two particular tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 403 diff changeset	2060 assert(p[-1] == '0' \|\| p[-1] == '1' \|\| p[-1] == '_' \|\| p[-1] == 'b' \|\| p[-1] == 'B', p[-1] ~ "");
367 dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module. aziz parents: 365 diff changeset	2061 assert( !(p == '0' \|\| p == '1' \|\| *p == '_') );
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2062 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2063
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2064 LscanOctal:
0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2065 assert(*p == '_');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2066 while (1)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2067 {
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2068 if (*++p == '_')
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2069 continue;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2070 if (!isoctal(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2071 break;
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2072 Loctal_enter_loop:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2073 if (ulong_ < ulong.max/2 \|\| (ulong_ == ulong.max/2 && *p <= '1'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2074 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2075 ulong_ *= 8;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2076 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2077 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2078 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2079 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2080 overflow = true;
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2081 while (isoctal(*++p)) {}
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2082 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2083 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2084
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2085 bool hasDecimalDigits;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2086 if (isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2087 {
396 0a4619735ce9 Applied fixes to Lexer, Parser and other classes. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 393 diff changeset	2088 Loctal_hasDecimalDigits:
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2089 hasDecimalDigits = true;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2090 while (isdigit(*++p)) {}
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2091 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2092
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2093 // The number could be a float, so check errors below.
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2094 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2095 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2096 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2097 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2098 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2099 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2100 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2101 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2102 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2103 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2104 goto LscanReal;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2105 default:
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2106 }
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2107
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2108 if (hasDecimalDigits)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2109 error(t.start, MID.OctalNumberHasDecimals);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2110
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2111 if (overflow)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2112 error(t.start, MID.OverflowOctalNumber);
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2113 // goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2114
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2115 Lfinalize:
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2116 enum Suffix
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2117 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2118 None = 0,
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2119 Unsigned = 1,
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2120 Long = 2
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2121 }
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2122
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2123 // Scan optional suffix: L, Lu, LU, u, uL, U or UL.
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2124 Suffix suffix;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2125 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2126 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2127 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2128 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2129 case 'L':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2130 if (suffix & Suffix.Long)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2131 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2132 suffix \|= Suffix.Long;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2133 ++p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2134 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2135 case 'u', 'U':
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2136 if (suffix & Suffix.Unsigned)
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2137 break;
60 32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2138 suffix \|= Suffix.Unsigned;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2139 ++p;
32cc23bd217b - Fixed number suffix scanning. aziz parents: 59 diff changeset	2140 continue;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2141 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2142 break;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2143 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2144 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2145 }
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2146
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2147 // Determine type of Integer.
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2148 switch (suffix)
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2149 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2150 case Suffix.None:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2151 if (ulong_ & 0x8000_0000_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2152 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2153 if (isDecimal)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2154 error(t.start, MID.OverflowDecimalSign);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2155 t.kind = TOK.Uint64;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2156 }
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2157 else if (ulong_ & 0xFFFF_FFFF_0000_0000)
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2158 t.kind = TOK.Int64;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2159 else if (ulong_ & 0x8000_0000)
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2160 t.kind = isDecimal ? TOK.Int64 : TOK.Uint32;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2161 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2162 t.kind = TOK.Int32;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2163 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2164 case Suffix.Unsigned:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2165 if (ulong_ & 0xFFFF_FFFF_0000_0000)
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2166 t.kind = TOK.Uint64;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2167 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2168 t.kind = TOK.Uint32;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2169 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2170 case Suffix.Long:
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2171 if (ulong_ & 0x8000_0000_0000_0000)
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2172 {
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2173 if (isDecimal)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2174 error(t.start, MID.OverflowDecimalSign);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2175 t.kind = TOK.Uint64;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2176 }
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2177 else
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2178 t.kind = TOK.Int64;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2179 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2180 case Suffix.Unsigned \| Suffix.Long:
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2181 t.kind = TOK.Uint64;
58 50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2182 break;
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2183 default:
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2184 assert(0);
50bb7fc9db44 - The types of integers are recognized now. aziz parents: 57 diff changeset	2185 }
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2186 t.ulong_ = ulong_;
15 c70c028e47dd - Started implementation of lexing numbers. aziz parents: 14 diff changeset	2187 t.end = p;
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2188 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2189 LscanReal:
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2190 scanReal(t);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2191 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2192 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2193
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2194 /// Scans a floating point number literal.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2195 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2196 /// $(PRE
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2197 /// FloatLiteral := Float[fFL]?i?
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2198 /// Float := DecFloat \| HexFloat
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2199 /// DecFloat := ([0-9][0-9_][.][0-9_]DecExponent?) \|
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2200 /// [.][0-9][0-9_]DecExponent? \| [0-9][0-9_]DecExponent
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2201 /// DecExponent := [eE][+-]?[0-9][0-9_]*
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2202 /// HexFloat := 0[xX](HexDigits[.]HexDigits \|
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2203 /// [.][0-9a-zA-Z]HexDigits? \|
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2204 /// HexDigits)HexExponent
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2205 /// HexExponent := [pP][+-]?[0-9][0-9_]*
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2206 /// )
56 63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2207 void scanReal(ref Token t)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers. aziz parents: 55 diff changeset	2208 {
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2209 if (*p == '.')
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2210 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2211 assert(p[1] != '.');
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2212 // This function was called by scan() or scanNumber().
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2213 while (isdigit(++p) \|\| p == '_') {}
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2214 }
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2215 else
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2216 // This function was called by scanNumber().
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2217 assert(delegate ()
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2218 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2219 switch (*p)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2220 {
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2221 case 'L':
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2222 if (p[1] != 'i')
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2223 return false;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2224 case 'i', 'f', 'F', 'e', 'E':
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2225 return true;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2226 default:
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2227 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2228 return false;
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2229 }()
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2230 );
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2231
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2232 // Scan exponent.
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2233 if (p == 'e' \|\| p == 'E')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2234 {
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2235 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2236 if (p == '-' \|\| p == '+')
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2237 ++p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2238 if (isdigit(*p))
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2239 while (isdigit(++p) \|\| p == '_') {}
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2240 else
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2241 error(t.start, MID.FloatExpMustStartWithDigit);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2242 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2243
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2244 // Copy whole number and remove underscores from buffer.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2245 char[] buffer = t.start[0..p-t.start].dup;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2246 uint j;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2247 foreach (c; buffer)
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2248 if (c != '_')
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2249 buffer[j++] = c;
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2250 buffer.length = j; // Adjust length.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2251 buffer ~= 0; // Terminate for C functions.
bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2252
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2253 finalizeFloat(t, buffer);
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2254 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2255
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2256 /// Scans a hexadecimal floating point number literal.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2257 void scanHexReal(ref Token t)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2258 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2259 assert(p == '.' \|\| p == 'p' \|\| *p == 'P');
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2260 MID mid;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2261 if (*p == '.')
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2262 while (ishexad(++p) \|\| p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2263 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2264 // Decimal exponent is required.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2265 if (p != 'p' && p != 'P')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2266 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2267 mid = MID.HexFloatExponentRequired;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2268 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2269 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2270 // Scan exponent
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2271 assert(p == 'p' \|\| p == 'P');
393 fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2272 ++p;
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2273 if (p == '+' \|\| p == '-')
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2274 ++p;
fce1e6133dac Applied fix to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 392 diff changeset	2275 if (!isdigit(*p))
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2276 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2277 mid = MID.HexFloatExpMustStartWithDigit;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2278 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2279 }
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2280 while (isdigit(++p) \|\| p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2281 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2282 // Copy whole number and remove underscores from buffer.
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2283 char[] buffer = t.start[0..p-t.start].dup;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2284 uint j;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2285 foreach (c; buffer)
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2286 if (c != '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2287 buffer[j++] = c;
c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2288 buffer.length = j; // Adjust length.
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2289 buffer ~= 0; // Terminate for C functions.
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2290 finalizeFloat(t, buffer);
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2291 return;
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2292 Lerr:
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2293 t.kind = TOK.Float32;
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2294 t.end = p;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2295 error(t.start, mid);
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2296 }
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2297
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2298 /// Sets the value of the token.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2299 /// Params:
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2300 /// t = receives the value.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2301 /// buffer = the well-formed float number.
63 c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2302 void finalizeFloat(ref Token t, string buffer)
c29229fbf2f7 - Recognizing floats that start with a dot. aziz parents: 62 diff changeset	2303 {
389 c4bfceab7246 Applied fixes and improvements to hex float scanner. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 388 diff changeset	2304 assert(buffer[$-1] == 0);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2305 // Float number is well-formed. Check suffixes and do conversion.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2306 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2307 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2308 case 'f', 'F':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2309 t.kind = TOK.Float32;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2310 t.float_ = strtof(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2311 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2312 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2313 case 'L':
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2314 t.kind = TOK.Float80;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2315 t.real_ = strtold(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2316 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2317 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2318 default:
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2319 t.kind = TOK.Float64;
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2320 t.double_ = strtod(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2321 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2322 if (*p == 'i')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2323 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2324 ++p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2325 t.kind += 3; // Switch to imaginary counterpart.
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2326 assert(t.kind == TOK.Imaginary32 \|\|
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2327 t.kind == TOK.Imaginary64 \|\|
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2328 t.kind == TOK.Imaginary80);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2329 }
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	2330 if (errno() == ERANGE)
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2331 error(t.start, MID.OverflowFloatNumber);
62 96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2332 t.end = p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2333 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block. aziz parents: 61 diff changeset	2334
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2335 /// Scans a special token sequence.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2336 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2337 /// SpecialTokenSequence := "#line" Integer Filespec? EndOfLine
360 b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence(). aziz parents: 350 diff changeset	2338 void scanSpecialTokenSequence(ref Token t)
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2339 {
47 8aa37a78937b - Properly implemented scanner for normal string literals. aziz parents: 46 diff changeset	2340 assert(*p == '#');
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2341 t.kind = TOK.HashLine;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	2342 t.setWhitespaceFlag();
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2343
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2344 MID mid;
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2345 char* errorAtColumn = p;
763 f26f13b5a3a3 Fixed code in scanSpecialTokenSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 762 diff changeset	2346 char* tokenEnd = ++p;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2347
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2348 if (!(p[0] == 'l' && p[1] == 'i' && p[2] == 'n' && p[3] == 'e'))
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2349 {
370 ae4afb66768f - Renamed findModule() to findModulePath(). aziz parents: 367 diff changeset	2350 mid = MID.ExpectedIdentifierSTLine;
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2351 goto Lerr;
cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2352 }
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2353 p += 3;
763 f26f13b5a3a3 Fixed code in scanSpecialTokenSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 762 diff changeset	2354 tokenEnd = p + 1;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2355
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2356 // TODO: #line58"path/file" is legal. Require spaces?
ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2357 // State.Space could be used for that purpose.
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2358 enum State
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2359 { /+Space,+/ Integer, Filespec, End }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2360
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2361 State state = State.Integer;
763 f26f13b5a3a3 Fixed code in scanSpecialTokenSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 762 diff changeset	2362
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2363 while (!isEndOfLine(++p))
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2364 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2365 if (isspace(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2366 continue;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2367 if (state == State.Integer)
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2368 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2369 if (!isdigit(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2370 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2371 errorAtColumn = p;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2372 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2373 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2374 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2375 t.tokLineNum = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2376 scan(*t.tokLineNum);
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2377 tokenEnd = p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2378 if (t.tokLineNum.kind != TOK.Int32 && t.tokLineNum.kind != TOK.Uint32)
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2379 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2380 errorAtColumn = t.tokLineNum.start;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2381 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2382 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2383 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2384 --p; // Go one back because scan() advanced p past the integer.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2385 state = State.Filespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2386 }
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2387 else if (state == State.Filespec && *p == '"')
763 f26f13b5a3a3 Fixed code in scanSpecialTokenSequence(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 762 diff changeset	2388 { // MID.ExpectedFilespec is deprecated.
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2389 // if (*p != '"')
c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2390 // {
c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2391 // errorAtColumn = p;
c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2392 // mid = MID.ExpectedFilespec;
c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2393 // goto Lerr;
c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2394 // }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2395 t.tokLineFilespec = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2396 t.tokLineFilespec.start = p;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2397 t.tokLineFilespec.kind = TOK.Filespec;
552 3bc7801c207e Refactored the way how tokens are flagged as whitespace. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 550 diff changeset	2398 t.tokLineFilespec.setWhitespaceFlag();
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2399 while (*++p != '"')
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2400 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2401 if (isEndOfLine(p))
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2402 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2403 errorAtColumn = t.tokLineFilespec.start;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2404 mid = MID.UnterminatedFilespec;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2405 t.tokLineFilespec.end = p;
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2406 tokenEnd = p;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2407 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2408 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2409 isascii(*p) \|\| decodeUTF8();
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2410 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2411 auto start = t.tokLineFilespec.start +1; // +1 skips '"'
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2412 t.tokLineFilespec.str = start[0 .. p - start];
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2413 t.tokLineFilespec.end = p + 1;
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2414 tokenEnd = p + 1;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2415 state = State.End;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2416 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2417 else/+ if (state == State.End)+/
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2418 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2419 mid = MID.UnterminatedSpecialToken;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2420 goto Lerr;
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2421 }
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2422 }
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2423 assert(isEndOfLine(p));
323 6259fb93e3dd - Rewrote scanSpecialToken(). aziz parents: 315 diff changeset	2424
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2425 if (state == State.Integer)
51 cadd2bfe686c - Displaying error messages in XML. aziz parents: 50 diff changeset	2426 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2427 errorAtColumn = p;
388 ae154eceba65 Applied some fixes to scanning and printing #line tokens. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 387 diff changeset	2428 mid = MID.ExpectedIntegerAfterSTLine;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2429 goto Lerr;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2430 }
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2431
392 bb935c6f9b7a Applied fixes and improvements to the Lexer class. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 391 diff changeset	2432 // Evaluate #line only when not in token string.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2433 if (!inTokenString && t.tokLineNum)
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2434 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2435 this.lineNum_hline = this.lineNum - t.tokLineNum.uint_ + 1;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2436 if (t.tokLineFilespec)
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	2437 newFilePath(t.tokLineFilespec.str);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2438 }
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2439 p = tokenEnd;
c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2440 t.end = tokenEnd;
30 426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2441
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2442 return;
426767b94635 - Added code for parsing the '#line' special token. aziz parents: 29 diff changeset	2443 Lerr:
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2444 p = tokenEnd;
c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2445 t.end = tokenEnd;
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2446 error(errorAtColumn, mid);
371 01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	2447 }
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer. aziz parents: 370 diff changeset	2448
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2449 /// Inserts an empty dummy token (TOK.Empty) before t.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2450 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2451 /// Useful in the parsing phase for representing a node in the AST
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2452 /// that doesn't consume an actual token from the source text.
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2453 Token* insertEmptyTokenBefore(Token* t)
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2454 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2455 assert(t !is null && t.prev !is null);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2456 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2457 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2458
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2459 auto prev_t = t.prev;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2460 auto new_t = new Token;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2461 new_t.kind = TOK.Empty;
414 9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2462 new_t.start = new_t.end = prev_t.end;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2463 // Link in new token.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2464 prev_t.next = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2465 new_t.prev = prev_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2466 new_t.next = t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2467 t.prev = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2468 return new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 413 diff changeset	2469 }
105 df34ec47fb81 - Added getState() method and State struct to Lexer. aziz parents: 103 diff changeset	2470
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2471 /// Returns the error line number.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2472 uint errorLineNumber(uint lineNum)
239 7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK. aziz parents: 223 diff changeset	2473 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2474 return lineNum - this.lineNum_hline;
18 c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	2475 }
c48d2125f1e2 - Moved code for scanning character literals to separate function. aziz parents: 17 diff changeset	2476
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2477 /// Forwards error parameters.
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2478 void error(char* columnPos, MID mid, ...)
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2479 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2480 error_(this.lineNum, this.lineBegin, columnPos, mid, _arguments, _argptr);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2481 }
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2482
798 c24be8d4f6ab Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 797 diff changeset	2483 /// ditto
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2484 void error(uint lineNum, char* lineBegin, char* columnPos, MID mid, ...)
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2485 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2486 error_(lineNum, lineBegin, columnPos, mid, _arguments, _argptr);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2487 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2488
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2489 /// Creates an error report and appends it to a list.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2490 /// Params:
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2491 /// lineNum = the line number.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2492 /// lineBegin = points to the first character of the current line.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2493 /// columnPos = points to the character where the error is located.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2494 /// mid = the message ID.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2495 void error_(uint lineNum, char* lineBegin, char* columnPos, MID mid,
762 c909a3d3fa52 Fixed vararg issue with gdc. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 755 diff changeset	2496 TypeInfo[] _arguments, Arg _argptr)
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2497 {
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2498 lineNum = this.errorLineNumber(lineNum);
607 2ed1e6d638cd Making use of struct NewlineData. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 601 diff changeset	2499 auto errorPath = this.filePaths.setPath;
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2500 auto location = new Location(errorPath, lineNum, lineBegin, columnPos);
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2501 auto msg = Format(_arguments, _argptr, GetMsg(mid));
520 f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	2502 auto error = new LexerError(location, msg);
f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	2503 errors ~= error;
f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	2504 if (infoMan !is null)
f203c5248d0b Added 'compile' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 519 diff changeset	2505 infoMan ~= error;
207 481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2506 }
481ed2b63a49 - Added contracts to method scan(). aziz parents: 131 diff changeset	2507
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2508 /// Scans the whole source text until EOF is encountered.
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2509 void scanAll()
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2510 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2511 while (nextToken() != TOK.EOF)
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2512 {}
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2513 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2514
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2515 /// Returns the first token of the source text.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2516 /// This can be the EOF token.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2517 /// Structure: HEAD -> Newline -> First Token
487 bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2518 Token* firstToken()
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2519 {
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2520 return this.head.next.next;
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2521 }
bccca748d745 Added 'tokenize' command. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 485 diff changeset	2522
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2523 /// Returns true if str is a valid D identifier.
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2524 static bool isIdentifierString(char[] str)
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2525 {
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2526 if (str.length == 0 \|\| isdigit(str[0]))
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2527 return false;
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2528 size_t idx;
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2529 do
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2530 {
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2531 auto c = dil.Unicode.decode(str, idx);
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2532 if (c == ERROR_CHAR \|\| !(isident(c) \|\| !isascii(c) && isUniAlpha(c)))
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2533 return false;
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2534 } while (idx < str.length)
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2535 return true;
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2536 }
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2537
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2538 /// Returns true if str is a keyword or a special token (__FILE__, __LINE__ etc.)
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2539 static bool isReservedIdentifier(char[] str)
dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2540 {
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2541 if (!isIdentifierString(str))
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2542 return false; // str is not a valid identifier.
382 66477017cb95 Applied fixes to isNonReservedIdentifier(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 371 diff changeset	2543
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2544 auto id = IdTable.inStatic(str);
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2545 if (id is null \|\| id.kind == TOK.Identifier)
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2546 return false; // str is not in the table or a normal identifier.
365 ed67acc82268 - Added option includes to config.d. aziz parents: 360 diff changeset	2547
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2548 return true;
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2549 }
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2550
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2551 /// Returns true if the current character to be decoded is
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2552 /// a Unicode alpha character.
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2553 ///
cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2554 /// The current pointer 'p' is not advanced if false is returned.
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2555 bool isUnicodeAlpha()
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2556 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2557 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2558 char* p = this.p;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2559 dchar d = *p;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2560 ++p; // Move to second byte.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2561 // Error if second byte is not a trail byte.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2562 if (!isTrailByte(*p))
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2563 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2564 // Check for overlong sequences.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2565 switch (d)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2566 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2567 case 0xE0, 0xF0, 0xF8, 0xFC:
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2568 if ((*p & d) == 0x80)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2569 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2570 default:
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2571 if ((d & 0xFE) == 0xC0) // 1100000x
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2572 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2573 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2574 const char[] checkNextByte = "if (!isTrailByte(*++p))"
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2575 " return false;";
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2576 const char[] appendSixBits = "d = (d << 6) \| *p & 0b0011_1111;";
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2577 // Decode
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2578 if ((d & 0b1110_0000) == 0b1100_0000)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2579 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2580 d &= 0b0001_1111;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2581 mixin(appendSixBits);
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2582 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2583 else if ((d & 0b1111_0000) == 0b1110_0000)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2584 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2585 d &= 0b0000_1111;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2586 mixin(appendSixBits ~
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2587 checkNextByte ~ appendSixBits);
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2588 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2589 else if ((d & 0b1111_1000) == 0b1111_0000)
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2590 {
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2591 d &= 0b0000_0111;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2592 mixin(appendSixBits ~
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2593 checkNextByte ~ appendSixBits ~
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2594 checkNextByte ~ appendSixBits);
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2595 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2596 else
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2597 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2598
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2599 assert(isTrailByte(*p));
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2600 if (!isValidChar(d) \|\| !isUniAlpha(d))
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2601 return false;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2602 // Only advance pointer if this is a Unicode alpha character.
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2603 this.p = p;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2604 return true;
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2605 }
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2606
082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2607 /// Decodes the next UTF-8 sequence.
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2608 dchar decodeUTF8()
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2609 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2610 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2611 char* p = this.p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2612 dchar d = *p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2613
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2614 ++p; // Move to second byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2615 // Error if second byte is not a trail byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2616 if (!isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2617 goto Lerr2;
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2618
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2619 // Check for overlong sequences.
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2620 switch (d)
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2621 {
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2622 case 0xE0, // 11100000 100xxxxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2623 0xF0, // 11110000 1000xxxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2624 0xF8, // 11111000 10000xxx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2625 0xFC: // 11111100 100000xx
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2626 if ((*p & d) == 0x80)
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2627 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2628 default:
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2629 if ((d & 0xFE) == 0xC0) // 1100000x
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2630 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2631 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2632
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2633 const char[] checkNextByte = "if (!isTrailByte(*++p))"
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2634 " goto Lerr2;";
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2635 const char[] appendSixBits = "d = (d << 6) \| *p & 0b0011_1111;";
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2636
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2637 // Decode
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2638 if ((d & 0b1110_0000) == 0b1100_0000)
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2639 { // 110xxxxx 10xxxxxx
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2640 d &= 0b0001_1111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2641 mixin(appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2642 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2643 else if ((d & 0b1111_0000) == 0b1110_0000)
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2644 { // 1110xxxx 10xxxxxx 10xxxxxx
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2645 d &= 0b0000_1111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2646 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2647 checkNextByte ~ appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2648 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2649 else if ((d & 0b1111_1000) == 0b1111_0000)
550 082126d78f90 Fixed lexing of identifiers. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 538 diff changeset	2650 { // 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2651 d &= 0b0000_0111;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2652 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2653 checkNextByte ~ appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2654 checkNextByte ~ appendSixBits);
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2655 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2656 else
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2657 // 5 and 6 byte UTF-8 sequences are not allowed yet.
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2658 // 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2659 // 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2660 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2661
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2662 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2663
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2664 if (!isValidChar(d))
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2665 {
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2666 Lerr:
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2667 // Three cases:
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2668 // *) the UTF-8 sequence was successfully decoded but the resulting
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2669 // character is invalid.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2670 // p points to last trail byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2671 // *) the UTF-8 sequence is overlong.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2672 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2673 // *) the UTF-8 sequence has more than 4 bytes or starts with
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2674 // a trail byte.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2675 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2676 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2677 // Move to next ASCII character or lead byte of a UTF-8 sequence.
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2678 while (p < (end-1) && isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2679 ++p;
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2680 --p;
426 3f7790d3f9d6 Improved decodeUTF8(). Aziz K?ksal <aziz.koeksal@gmail.com> parents: 425 diff changeset	2681 assert(!isTrailByte(p[1]));
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2682 Lerr2:
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2683 d = REPLACEMENT_CHAR;
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2684 error(this.p, MID.InvalidUTF8Sequence, formatBytes(this.p, p));
425 6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2685 }
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2686
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2687 this.p = p;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2688 return d;
6bf936bf3356 Added own UTF-8 decoding function. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 424 diff changeset	2689 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2690
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2691 /// Encodes the character d and appends it to str.
519 9ebc799c7dc5 Fixes in dil.Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 514 diff changeset	2692 static void encodeUTF8(ref char[] str, dchar d)
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2693 {
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2694 assert(!isascii(d), "check for ASCII char before calling encodeUTF8().");
510 dd3ce87b3569 Added module dil.Unicode. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 507 diff changeset	2695 assert(isValidChar(d), "check if character is valid before calling encodeUTF8().");
430 e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 426 diff changeset	2696
519 9ebc799c7dc5 Fixes in dil.Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 514 diff changeset	2697 char[6] b = void;
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2698 if (d < 0x800)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2699 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2700 b[0] = 0xC0 \| (d >> 6);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2701 b[1] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2702 str ~= b[0..2];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2703 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2704 else if (d < 0x10000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2705 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2706 b[0] = 0xE0 \| (d >> 12);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2707 b[1] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2708 b[2] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2709 str ~= b[0..3];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2710 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2711 else if (d < 0x200000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2712 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2713 b[0] = 0xF0 \| (d >> 18);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2714 b[1] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2715 b[2] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2716 b[3] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2717 str ~= b[0..4];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2718 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2719 /+ // There are no 5 and 6 byte UTF-8 sequences yet.
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2720 else if (d < 0x4000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2721 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2722 b[0] = 0xF8 \| (d >> 24);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2723 b[1] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2724 b[2] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2725 b[3] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2726 b[4] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2727 str ~= b[0..5];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2728 }
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2729 else if (d < 0x80000000)
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2730 {
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2731 b[0] = 0xFC \| (d >> 30);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2732 b[1] = 0x80 \| ((d >> 24) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2733 b[2] = 0x80 \| ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2734 b[3] = 0x80 \| ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2735 b[4] = 0x80 \| ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2736 b[5] = 0x80 \| (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2737 str ~= b[0..6];
c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2738 }
424 bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2739 +/
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2740 else
bb3cb00feeb2 Applied some fixes to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 422 diff changeset	2741 assert(0);
48 c2e0e0269c28 - Added code for scanning escape string literals. aziz parents: 47 diff changeset	2742 }
789 c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2743
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2744 /// Formats the bytes between start and end.
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2745 /// Returns: e.g.: abc -> \x61\x62\x63
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2746 static char[] formatBytes(char* start, char* end)
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2747 {
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2748 auto strLen = end-start;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2749 const formatLen = `\xXX`.length;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2750 char[] result = new char[strLen*formatLen]; // Reserve space.
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2751 result.length = 0;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2752 foreach (c; cast(ubyte[])start[0..strLen])
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2753 result ~= Format("\\x{:X}", c);
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2754 return result;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2755 }
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2756
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2757 /// Searches for an invalid UTF-8 sequence in str.
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2758 /// Returns: a formatted string of the invalid sequence (e.g. \xC0\x80).
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2759 static string findInvalidUTF8Sequence(string str)
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2760 {
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2761 char* p = str.ptr, end = p + str.length;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2762 while (p < end)
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2763 {
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2764 if (decode(p, end) == ERROR_CHAR)
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2765 {
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2766 auto begin = p;
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2767 // Skip trail-bytes.
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2768 while (++p < end && isTrailByte(*p))
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2769 {}
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2770 return Lexer.formatBytes(begin, p);
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2771 }
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2772 }
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2773 assert(p == end);
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2774 return "";
c1d5cfd7aa44 Implemented string literal conversion. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 776 diff changeset	2775 }
5 79b4e8848794 - Started writing XML generator. aziz parents: 4 diff changeset	2776 }
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2777
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2778 /// Tests the lexer with a list of tokens.
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2779 unittest
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2780 {
391 33b566df6af4 Migrated project to Tango. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 390 diff changeset	2781 Stdout("Testing Lexer.\n");
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2782 struct Pair
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2783 {
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2784 char[] tokenText;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2785 TOK kind;
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2786 }
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2787 static Pair[] pairs = [
494 9a7ca8c56e59 Refactored a few things in the Parser. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 490 diff changeset	2788 {"#!äöüß", TOK.Shebang}, {"\n", TOK.Newline},
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2789 {"//çay", TOK.Comment}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2790 {"&", TOK.AndBinary},
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2791 {"/çağ/", TOK.Comment}, {"&&", TOK.AndLogical},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2792 {"/+çak+/", TOK.Comment}, {"&=", TOK.AndAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2793 {">", TOK.Greater}, {"+", TOK.Plus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2794 {">=", TOK.GreaterEqual}, {"++", TOK.PlusPlus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2795 {">>", TOK.RShift}, {"+=", TOK.PlusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2796 {">>=", TOK.RShiftAssign}, {"-", TOK.Minus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2797 {">>>", TOK.URShift}, {"--", TOK.MinusMinus},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2798 {">>>=", TOK.URShiftAssign}, {"-=", TOK.MinusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2799 {"<", TOK.Less}, {"=", TOK.Assign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2800 {"<=", TOK.LessEqual}, {"==", TOK.Equal},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2801 {"<>", TOK.LorG}, {"~", TOK.Tilde},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2802 {"<>=", TOK.LorEorG}, {"~=", TOK.CatAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2803 {"<<", TOK.LShift}, {"*", TOK.Mul},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2804 {"<<=", TOK.LShiftAssign}, {"*=", TOK.MulAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2805 {"!", TOK.Not}, {"/", TOK.Div},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2806 {"!=", TOK.NotEqual}, {"/=", TOK.DivAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2807 {"!<", TOK.UorGorE}, {"^", TOK.Xor},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2808 {"!>", TOK.UorLorE}, {"^=", TOK.XorAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2809 {"!<=", TOK.UorG}, {"%", TOK.Mod},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2810 {"!>=", TOK.UorL}, {"%=", TOK.ModAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2811 {"!<>", TOK.UorE}, {"(", TOK.LParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2812 {"!<>=", TOK.Unordered}, {")", TOK.RParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2813 {".", TOK.Dot}, {"[", TOK.LBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2814 {"..", TOK.Slice}, {"]", TOK.RBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2815 {"...", TOK.Ellipses}, {"{", TOK.LBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2816 {"\|", TOK.OrBinary}, {"}", TOK.RBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2817 {"\|\|", TOK.OrLogical}, {":", TOK.Colon},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2818 {"\|=", TOK.OrAssign}, {";", TOK.Semicolon},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2819 {"?", TOK.Question}, {",", TOK.Comma},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2820 {"$", TOK.Dollar}, {"cam", TOK.Identifier},
0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2821 {"çay", TOK.Identifier}, {".0", TOK.Float64},
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2822 {"0", TOK.Int32}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2823 {"\r", TOK.Newline}, {"\r\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2824 {"\u2028", TOK.Newline}, {"\u2029", TOK.Newline}
40 9d5ceb0f8be9 - Added more tokens for testing. aziz parents: 39 diff changeset	2825 ];
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2826
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2827 char[] src;
69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2828
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2829 // Join all token texts into a single string.
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2830 foreach (i, pair; pairs)
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2831 if (pair.kind == TOK.Comment && pair.tokenText[1] == '/' \|\| // Line comment.
ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2832 pair.kind == TOK.Shebang)
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2833 {
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2834 assert(pairs[i+1].kind == TOK.Newline); // Must be followed by a newline.
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2835 src ~= pair.tokenText;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2836 }
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2837 else
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2838 src ~= pair.tokenText ~ " ";
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2839
764 4579e8505d5e Fixed unittests and removed dil.File. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 763 diff changeset	2840 auto lx = new Lexer(new SourceText("", src));
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2841 auto token = lx.getTokens();
39 69b940398d7b - Added unittest to test correct parsing of operator tokens. aziz parents: 38 diff changeset	2842
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2843 uint i;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2844 assert(token == lx.head);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2845 assert(token.next.kind == TOK.Newline);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2846 token = token.next.next;
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2847 do
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2848 {
413 0fd78fdcb982 Added an alternative scan() method to class Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 412 diff changeset	2849 assert(i < pairs.length);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2850 assert(token.srcText == pairs[i].tokenText, Format("Scanned '{0}' but expected '{1}'", token.srcText, pairs[i].tokenText));
350 4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2851 ++i;
4ea6759300cf - Fixed unittests in module Lexer. aziz parents: 346 diff changeset	2852 token = token.next;
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2853 } while (token.kind != TOK.EOF)
41 2b7be1d67d4d - Optimized scanner of block comments. aziz parents: 40 diff changeset	2854 }
55 5887751f8e04 - Relocated ptable to the bottom of the source file. aziz parents: 54 diff changeset	2855
797 cf2ad5df025c Added documentation comments. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 789 diff changeset	2856 /// Tests the Lexer's peek() method.
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2857 unittest
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2858 {
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2859 Stdout("Testing method Lexer.peek()\n");
764 4579e8505d5e Fixed unittests and removed dil.File. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 763 diff changeset	2860 auto sourceText = new SourceText("", "unittest { }");
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2861 auto lx = new Lexer(sourceText, null);
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2862
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2863 auto next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2864 lx.peek(next);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2865 assert(next.kind == TOK.Newline);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2866 lx.peek(next);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2867 assert(next.kind == TOK.Unittest);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2868 lx.peek(next);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2869 assert(next.kind == TOK.LBrace);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2870 lx.peek(next);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2871 assert(next.kind == TOK.RBrace);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2872 lx.peek(next);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2873 assert(next.kind == TOK.EOF);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2874
764 4579e8505d5e Fixed unittests and removed dil.File. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 763 diff changeset	2875 lx = new Lexer(new SourceText("", ""));
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2876 next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2877 lx.peek(next);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2878 assert(next.kind == TOK.Newline);
485 ea8c7459f1c4 Changed a lot of things in the Lexer. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 464 diff changeset	2879 lx.peek(next);
679 ff6971637f88 Renamed Token member type to kind. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 607 diff changeset	2880 assert(next.kind == TOK.EOF);
422 ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2881 }
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2882
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2883 unittest
ad7977fe315a Added support for column numbers in error messages. Aziz K?ksal <aziz.koeksal@gmail.com> parents: 420 diff changeset	2884 {
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2885 // Numbers unittest
68 7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2886 // 0L 0ULi 0_L 0_UL 0x0U 0x0p2 0_Fi 0_e2 0_F 0_i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2887 // 0u 0U 0uL 0UL 0L 0LU 0Lu
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2888 // 0Li 0f 0F 0fi 0Fi 0i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2889 // 0b_1_LU 0b1000u
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest. aziz parents: 67 diff changeset	2890 // 0x232Lu
67 996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem. aziz parents: 66 diff changeset	2891 }

Mercurial > projects > dil

annotate trunk/src/dil/lexer/Lexer.d @ 798:c24be8d4f6ab