Mercurial > projects > dil

annotate trunk/src/dil/lexer/Lexer.d @ 764:4579e8505d5e

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fixed unittests and removed dil.File. Fixed Converter.UTF16toUTF8(). Fixed an encode() function in dil.Unicode.
author Aziz K?ksal <aziz.koeksal@gmail.com>
date Sat, 16 Feb 2008 03:28:39 +0100
parents f26f13b5a3a3
children 5e3ef1b2011c
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
1 /++
8ba2570de175 Initial import.
aziz
parents:
diff changeset
2 Author: Aziz Köksal
249
32d354584b28 - Upgraded license notices to GPL3.
aziz
parents: 239
diff changeset
3 License: GPL3
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
4 +/
576
0df647660e76 Moved Lexer.d to new package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 552
diff changeset
5 module dil.lexer.Lexer;
577
9e811db780a6 Moved LexerFuncs.d to package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 576
diff changeset
6
596
39fac5531b85 Moved dil.Token to dil.lexer.Token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
7 import dil.lexer.Token;
599
c4cdea3a65ba Moved dil.Keywords to dil.lexer.Keywords.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 596
diff changeset
8 import dil.lexer.Keywords;
600
041eae272362 Moved dil.Identifier to dil.lexer.Identifier.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 599
diff changeset
9 import dil.lexer.Identifier;
601
9f96fd51cb41 Moved dil.IdTable to dil.lexer.IdTable.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
10 import dil.lexer.IdTable;
327
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
11 import dil.Information;
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
12 import dil.Messages;
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
13 import dil.HtmlEntities;
464
325714d8aa6c Added new module with compiler constants to resolve import cycle.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 434
diff changeset
14 import dil.CompilerInfo;
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
15 import dil.Unicode;
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
16 import dil.SourceText;
737
f88b5285b86b Implemented DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 715
diff changeset
17 import common;
f88b5285b86b Implemented DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 715
diff changeset
18
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
19 import tango.stdc.stdlib : strtof, strtod, strtold;
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
20 import tango.stdc.errno : errno, ERANGE;
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
21 import tango.stdc.time : time_t, time, ctime;
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
22 import tango.stdc.string : strlen;
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
23
577
9e811db780a6 Moved LexerFuncs.d to package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 576
diff changeset
24 public import dil.lexer.Funcs;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
25
499
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
26 /++
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
27 The Lexer analyzes the characters of a source text and
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
28 produces a doubly-linked list of tokens.
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
29 +/
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
30 class Lexer
8ba2570de175 Initial import.
aziz
parents:
diff changeset
31 {
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
32 SourceText srcText; /// The source text.
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
33 char* p; /// Points to the current character in the source text.
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
34 char* end; /// Points one character past the end of the source text.
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
35
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
36 Token* head; /// The head of the doubly linked token list.
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
37 Token* tail; /// The tail of the linked list. Set in scan().
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
38 Token* token; /// Points to the current token in the token list.
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
39
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
40 // Members used for error messages:
532
50e64bab9c7a Renamed InformationManager to InfoManager.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 520
diff changeset
41 InfoManager infoMan;
514
6ddff941862a Added new error classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 510
diff changeset
42 LexerError[] errors;
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
43 /// Always points to the first character of the current line.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
44 char* lineBegin;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
45 // Token* newline; /// Current newline token.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
46 uint lineNum = 1; /// Current, actual source text line number.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
47 uint lineNum_hline; /// Line number set by #line.
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
48 uint inTokenString; /// > 0 if inside q{ }
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
49 NewlineData.FilePaths* filePaths;
103
511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too.
aziz
parents: 94
diff changeset
50
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
51 /++
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
52 Construct a Lexer object.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
53 Params:
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
54 srcText = the UTF-8 source code.
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
55 infoMan = used for collecting error messages.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
56 +/
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
57 this(SourceText srcText, InfoManager infoMan = null)
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
58 {
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
59 this.srcText = srcText;
520
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
60 this.infoMan = infoMan;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
61
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
62 assert(text.length && text[$-1] == 0, "source text has no sentinel character");
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
63 this.p = text.ptr;
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
64 this.end = this.p + text.length;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
65 this.lineBegin = this.p;
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
66
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
67 this.head = new Token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
68 this.head.kind = TOK.HEAD;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
69 this.head.start = this.head.end = this.p;
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
70 this.token = this.head;
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
71 // Initialize this.filePaths.
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
72 newFilePath(this.srcText.filePath);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
73 // Add a newline as the first token after the head.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
74 auto newline = new Token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
75 newline.kind = TOK.Newline;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
76 newline.setWhitespaceFlag();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
77 newline.start = newline.end = this.p;
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
78 newline.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
79 newline.newline.oriLineNum = 1;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
80 newline.newline.setLineNum = 0;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
81 // Link in.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
82 this.token.next = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
83 newline.prev = this.token;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
84 this.token = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
85 // this.newline = newline;
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
86 scanShebang();
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
87 }
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
88
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
89 ~this()
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
90 {
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
91 auto token = head.next;
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
92 while (token !is null)
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
93 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
94 assert(token.kind == TOK.EOF ? token == tail && token.next is null : 1);
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
95 delete token.prev;
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
96 token = token.next;
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
97 }
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
98 delete tail;
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
99 }
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
100
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
101 char[] text()
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
102 {
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
103 return srcText.data;
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
104 }
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
105
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
106 /++
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
107 The "shebang" may optionally appear once at the beginning of a file.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
108 Regexp: #![^\EndOfLine]*
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
109 +/
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
110 void scanShebang()
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
111 {
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
112 if (*p == '#' && p[1] == '!')
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
113 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
114 auto t = new Token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
115 t.kind = TOK.Shebang;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
116 t.setWhitespaceFlag();
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
117 t.start = p;
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
118 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
119 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
120 isascii(*p) || decodeUTF8();
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
121 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
122 this.token.next = t;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
123 t.prev = this.token;
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
124 }
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
125 }
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
126
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
127 void finalizeSpecialToken(ref Token t)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
128 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
129 assert(t.srcText[0..2] == "__");
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
130 switch (t.kind)
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
131 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
132 case TOK.FILE:
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
133 t.str = this.filePaths.setPath;
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
134 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
135 case TOK.LINE:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
136 t.uint_ = this.errorLineNumber(this.lineNum);
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
137 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
138 case TOK.DATE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
139 TOK.TIME,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
140 TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
141 time_t time_val;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
142 time(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
143 char* str = ctime(&time_val);
742
fc8f0e61bc42 Fixed building of time_str.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 737
diff changeset
144 char[] time_str = str[0 .. strlen(str)-1]; // -1 removes trailing '\n'.
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
145 switch (t.kind)
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
146 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
147 case TOK.DATE:
346
ce9e2e77743f - Fix: include one space when slicing time_str for __DATE__.
aziz
parents: 344
diff changeset
148 time_str = time_str[4..11] ~ time_str[20..24] ~ \0; break;
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
149 case TOK.TIME:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
150 time_str = time_str[11..19] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
151 case TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
152 time_str = time_str[0..24] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
153 default: assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
154 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
155 t.str = time_str;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
156 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
157 case TOK.VENDOR:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
158 t.str = VENDOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
159 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
160 case TOK.VERSION:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
161 t.uint_ = VERSION_MAJOR*1000 + VERSION_MINOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
162 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
163 default:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
164 assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
165 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
166 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
167
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
168 /// Sets a new file path.
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
169 void newFilePath(char[] newPath)
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
170 {
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
171 auto paths = new NewlineData.FilePaths;
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
172 paths.oriPath = this.srcText.filePath;
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
173 paths.setPath = newPath;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
174 this.filePaths = paths;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
175 }
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
176
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
177 private void setLineBegin(char* p)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
178 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
179 // Check that we can look behind one character.
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
180 assert((p-1) >= text.ptr && p < end);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
181 // Check that previous character is a newline.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
182 assert(isNewlineEnd(p - 1));
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
183 this.lineBegin = p;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
184 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
185
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
186 private void scanNext(ref Token* t)
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
187 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
188 assert(t !is null);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
189 if (t.next)
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
190 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
191 t = t.next;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
192 // if (t.kind == TOK.Newline)
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
193 // this.newline = t;
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
194 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
195 else if (t != this.tail)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
196 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
197 Token* new_t = new Token;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
198 scan(*new_t);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
199 new_t.prev = t;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
200 t.next = new_t;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
201 t = new_t;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
202 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
203 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
204
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
205 /// Advance t one token forward.
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
206 void peek(ref Token* t)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
207 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
208 scanNext(t);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
209 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
210
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
211 /// Advance to the next token in the source text.
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
212 TOK nextToken()
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
213 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
214 scanNext(this.token);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
215 return this.token.kind;
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
216 }
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
217
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
218 /// Returns true if p points to the last character of a Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
219 bool isNewlineEnd(char* p)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
220 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
221 if (*p == '\n' || *p == '\r')
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
222 return true;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
223 if (*p == LS[2] || *p == PS[2])
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
224 if ((p-2) >= text.ptr)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
225 if (p[-1] == LS[1] && p[-2] == LS[0])
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
226 return true;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
227 return false;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
228 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
229
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
230 /++
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
231 This is the old scan method.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
232 TODO: profile old and new to see which one is faster.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
233 +/
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
234 public void scan(ref Token t)
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
235 in
3
4bbce78bfb1e - Added TOK enum.
aziz
parents: 2
diff changeset
236 {
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
237 assert(text.ptr <= p && p < end);
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
238 }
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
239 out
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
240 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
241 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
242 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
243 }
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
244 body
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
245 {
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
246 // Scan whitespace.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
247 if (isspace(*p))
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
248 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
249 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
250 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
251 {}
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
252 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
253
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
254 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
255 uint c = *p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
256 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
257 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
258 // Newline.
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
259 switch (*p)
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
260 {
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
261 case '\r':
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
262 if (p[1] == '\n')
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
263 ++p;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
264 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
265 assert(isNewlineEnd(p));
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
266 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
267 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
268 setLineBegin(p);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
269 // this.newline = &t;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
270 t.kind = TOK.Newline;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
271 t.setWhitespaceFlag();
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
272 t.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
273 t.newline.oriLineNum = lineNum;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
274 t.newline.setLineNum = lineNum_hline;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
275 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
276 return;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
277 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
278 if (isUnicodeNewline(p))
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
279 {
412
fb31af0fda73 Added struct Location, and token2LocTable to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 411
diff changeset
280 ++p; ++p;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 411
diff changeset
281 goto case '\n';
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
282 }
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
283 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
284 // Identifier or string literal.
13
e5211758b63c - Added isidbeg() function.
aziz
parents: 12
diff changeset
285 if (isidbeg(c))
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
286 {
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
287 if (c == 'r' && p[1] == '"' && ++p)
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
288 return scanRawStringLiteral(t);
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
289 if (c == 'x' && p[1] == '"')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
290 return scanHexStringLiteral(t);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
291 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
292 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
293 if (c == 'q' && p[1] == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
294 return scanDelimitedStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
295 if (c == 'q' && p[1] == '{')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
296 return scanTokenStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
297 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
298 // Scan identifier.
12
0989206cf73c - Added code to decode Unicode characters in identifiers.
aziz
parents: 11
diff changeset
299 Lidentifier:
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
300 do
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
301 { c = *++p; }
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
302 while (isident(c) || !isascii(c) && isUnicodeAlpha())
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
303
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
304 t.end = p;
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
305
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
306 auto id = IdTable.lookup(t.srcText);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
307 t.kind = id.type;
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
308 t.ident = id;
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
309
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
310 if (t.kind == TOK.Identifier || t.isKeyword)
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
311 return;
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
312 else if (t.isSpecialToken)
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
313 finalizeSpecialToken(t);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
314 else if (t.kind == TOK.EOF)
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
315 {
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
316 tail = &t;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
317 assert(t.srcText == "__EOF__");
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
318 }
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
319 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
320 assert(0, "unexpected token type: " ~ Token.toString(t.kind));
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
321 return;
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
322 }
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
323
15
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
324 if (isdigit(c))
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
325 return scanNumber(t);
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
326
8
d4ba94a5a282 - Parsing /* */ comments now.
aziz
parents: 7
diff changeset
327 if (c == '/')
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
328 {
8
d4ba94a5a282 - Parsing /* */ comments now.
aziz
parents: 7
diff changeset
329 c = *++p;
14
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
330 switch(c)
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
331 {
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
332 case '=':
14
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
333 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
334 t.kind = TOK.DivAssign;
14
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
335 t.end = p;
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
336 return;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
337 case '+':
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
338 return scanNestedComment(t);
32
d7011daa4740 - Added missing commas after the items in the messages table.
aziz
parents: 31
diff changeset
339 case '*':
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
340 return scanBlockComment(t);
32
d7011daa4740 - Added missing commas after the items in the messages table.
aziz
parents: 31
diff changeset
341 case '/':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
342 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
343 isascii(*p) || decodeUTF8();
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
344 t.kind = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
345 t.setWhitespaceFlag();
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
346 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
347 return;
32
d7011daa4740 - Added missing commas after the items in the messages table.
aziz
parents: 31
diff changeset
348 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
349 t.kind = TOK.Div;
10
3ee65d6e39c9 - Parsing // comments now.
aziz
parents: 9
diff changeset
350 t.end = p;
3ee65d6e39c9 - Parsing // comments now.
aziz
parents: 9
diff changeset
351 return;
3ee65d6e39c9 - Parsing // comments now.
aziz
parents: 9
diff changeset
352 }
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
353 }
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
354
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
355 switch (c)
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
356 {
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
357 case '\'':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
358 return scanCharacterLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
359 case '`':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
360 return scanRawStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
361 case '"':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
362 return scanNormalStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
363 case '\\':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
364 char[] buffer;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
365 do
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
366 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
367 c = scanEscapeSequence();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
368 if (isascii(c))
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
369 buffer ~= c;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
370 else
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
371 encodeUTF8(buffer, c);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
372 } while (*p == '\\')
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
373 buffer ~= 0;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
374 t.kind = TOK.String;
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
375 t.str = buffer;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
376 t.end = p;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
377 return;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
378 case '>': /* > >= >> >>= >>> >>>= */
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
379 c = *++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
380 switch (c)
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
381 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
382 case '=':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
383 t.kind = TOK.GreaterEqual;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
384 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
385 case '>':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
386 if (p[1] == '>')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
387 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
388 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
389 if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
390 { ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
391 t.kind = TOK.URShiftAssign;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
392 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
393 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
394 t.kind = TOK.URShift;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
395 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
396 else if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
397 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
398 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
399 t.kind = TOK.RShiftAssign;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
400 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
401 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
402 t.kind = TOK.RShift;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
403 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
404 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
405 t.kind = TOK.Greater;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
406 goto Lcommon2;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
407 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
408 assert(0);
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
409 case '<': /* < <= <> <>= << <<= */
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
410 c = *++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
411 switch (c)
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
412 {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
413 case '=':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
414 t.kind = TOK.LessEqual;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
415 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
416 case '<':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
417 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
418 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
419 t.kind = TOK.LShiftAssign;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
420 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
421 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
422 t.kind = TOK.LShift;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
423 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
424 case '>':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
425 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
426 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
427 t.kind = TOK.LorEorG;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
428 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
429 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
430 t.kind = TOK.LorG;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
431 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
432 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
433 t.kind = TOK.Less;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
434 goto Lcommon2;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
435 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
436 assert(0);
37
7f3bcb97d017 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 36
diff changeset
437 case '!': /* ! !< !> !<= !>= !<> !<>= */
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
438 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
439 switch (c)
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
440 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
441 case '<':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
442 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
443 if (c == '>')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
444 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
445 if (p[1] == '=') {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
446 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
447 t.kind = TOK.Unordered;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
448 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
449 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
450 t.kind = TOK.UorE;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
451 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
452 else if (c == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
453 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
454 t.kind = TOK.UorG;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
455 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
456 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
457 t.kind = TOK.UorGorE;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
458 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
459 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
460 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
461 case '>':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
462 if (p[1] == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
463 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
464 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
465 t.kind = TOK.UorL;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
466 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
467 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
468 t.kind = TOK.UorLorE;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
469 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
470 case '=':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
471 t.kind = TOK.NotEqual;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
472 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
473 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
474 t.kind = TOK.Not;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
475 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
476 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
477 assert(0);
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
478 case '.': /* . .[0-9] .. ... */
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
479 if (p[1] == '.')
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
480 {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
481 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
482 if (p[1] == '.') {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
483 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
484 t.kind = TOK.Ellipses;
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
485 }
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
486 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
487 t.kind = TOK.Slice;
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
488 }
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
489 else if (isdigit(p[1]))
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
490 {
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
491 return scanReal(t);
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
492 }
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
493 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
494 t.kind = TOK.Dot;
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
495 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
496 case '|': /* | || |= */
23
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
497 c = *++p;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
498 if (c == '=')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
499 t.kind = TOK.OrAssign;
23
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
500 else if (c == '|')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
501 t.kind = TOK.OrLogical;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
502 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
503 t.kind = TOK.OrBinary;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
504 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
505 }
23
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
506 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
507 case '&': /* & && &= */
24
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
508 c = *++p;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
509 if (c == '=')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
510 t.kind = TOK.AndAssign;
24
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
511 else if (c == '&')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
512 t.kind = TOK.AndLogical;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
513 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
514 t.kind = TOK.AndBinary;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
515 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
516 }
24
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
517 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
518 case '+': /* + ++ += */
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
519 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
520 if (c == '=')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
521 t.kind = TOK.PlusAssign;
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
522 else if (c == '+')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
523 t.kind = TOK.PlusPlus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
524 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
525 t.kind = TOK.Plus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
526 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
527 }
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
528 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
529 case '-': /* - -- -= */
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
530 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
531 if (c == '=')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
532 t.kind = TOK.MinusAssign;
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
533 else if (c == '-')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
534 t.kind = TOK.MinusMinus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
535 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
536 t.kind = TOK.Minus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
537 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
538 }
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
539 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
540 case '=': /* = == */
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
541 if (p[1] == '=') {
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
542 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
543 t.kind = TOK.Equal;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
544 }
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
545 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
546 t.kind = TOK.Assign;
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
547 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
548 case '~': /* ~ ~= */
27
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
549 if (p[1] == '=') {
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
550 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
551 t.kind = TOK.CatAssign;
27
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
552 }
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
553 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
554 t.kind = TOK.Tilde;
27
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
555 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
556 case '*': /* * *= */
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
557 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
558 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
559 t.kind = TOK.MulAssign;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
560 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
561 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
562 t.kind = TOK.Mul;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
563 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
564 case '^': /* ^ ^= */
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
565 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
566 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
567 t.kind = TOK.XorAssign;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
568 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
569 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
570 t.kind = TOK.Xor;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
571 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
572 case '%': /* % %= */
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
573 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
574 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
575 t.kind = TOK.ModAssign;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
576 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
577 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
578 t.kind = TOK.Mod;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
579 goto Lcommon;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
580 // Single character tokens:
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
581 case '(':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
582 t.kind = TOK.LParen;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
583 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
584 case ')':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
585 t.kind = TOK.RParen;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
586 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
587 case '[':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
588 t.kind = TOK.LBracket;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
589 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
590 case ']':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
591 t.kind = TOK.RBracket;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
592 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
593 case '{':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
594 t.kind = TOK.LBrace;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
595 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
596 case '}':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
597 t.kind = TOK.RBrace;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
598 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
599 case ':':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
600 t.kind = TOK.Colon;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
601 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
602 case ';':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
603 t.kind = TOK.Semicolon;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
604 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
605 case '?':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
606 t.kind = TOK.Question;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
607 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
608 case ',':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
609 t.kind = TOK.Comma;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
610 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
611 case '$':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
612 t.kind = TOK.Dollar;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
613 Lcommon:
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
614 ++p;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
615 Lcommon2:
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
616 t.end = p;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
617 return;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
618 case '#':
360
b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence().
aziz
parents: 350
diff changeset
619 return scanSpecialTokenSequence(t);
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
620 default:
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
621 }
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
622
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
623 // Check for EOF
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
624 if (isEOF(c))
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
625 {
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
626 assert(isEOF(*p), ""~*p);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
627 t.kind = TOK.EOF;
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
628 t.end = p;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
629 tail = &t;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
630 assert(t.start == t.end);
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
631 return;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
632 }
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
633
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
634 if (!isascii(c))
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
635 {
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
636 c = decodeUTF8();
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
637 if (isUniAlpha(c))
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
638 goto Lidentifier;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
639 }
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
640
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
641 error(t.start, MID.IllegalCharacter, cast(dchar)c);
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
642
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
643 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
644 t.kind = TOK.Illegal;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
645 t.setWhitespaceFlag();
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
646 t.dchar_ = c;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
647 t.end = p;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
648 return;
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
649 }
3
4bbce78bfb1e - Added TOK enum.
aziz
parents: 2
diff changeset
650 }
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
651
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
652 template toUint(char[] T)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
653 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
654 static assert(0 < T.length && T.length <= 4);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
655 static if (T.length == 1)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
656 const uint toUint = T[0];
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
657 else
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
658 const uint toUint = (T[0] << ((T.length-1)*8)) | toUint!(T[1..$]);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
659 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
660 static assert(toUint!("\xAA\xBB\xCC\xDD") == 0xAABBCCDD);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
661
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
662 // Can't use this yet due to a bug in DMD (bug id=1534).
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
663 template case_(char[] str, TOK tok, char[] label)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
664 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
665 const char[] case_ =
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
666 `case `~toUint!(str).stringof~`:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
667
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
668 goto `~label~`;`;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
669 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
670
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
671 template case_L4(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
672 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
673 const char[] case_L4 = case_!(str, tok, "Lcommon_4");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
674 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
675
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
676 template case_L3(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
677 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
678 const char[] case_L3 = case_!(str, tok, "Lcommon_3");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
679 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
680
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
681 template case_L2(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
682 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
683 const char[] case_L2 = case_!(str, tok, "Lcommon_2");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
684 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
685
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
686 template case_L1(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
687 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
688 const char[] case_L3 = case_!(str, tok, "Lcommon");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
689 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
690
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
691 public void scan_(ref Token t)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
692 in
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
693 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
694 assert(text.ptr <= p && p < end);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
695 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
696 out
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
697 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
698 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
699 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
700 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
701 body
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
702 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
703 // Scan whitespace.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
704 if (isspace(*p))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
705 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
706 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
707 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
708 {}
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
709 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
710
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
711 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
712 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
713 // Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
714 switch (*p)
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
715 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
716 case '\r':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
717 if (p[1] == '\n')
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
718 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
719 case '\n':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
720 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
721 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
722 ++lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
723 setLineBegin(p);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
724 // this.newline = &t;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
725 t.kind = TOK.Newline;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
726 t.setWhitespaceFlag();
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
727 t.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
728 t.newline.oriLineNum = lineNum;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
729 t.newline.setLineNum = lineNum_hline;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
730 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
731 return;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
732 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
733 if (isUnicodeNewline(p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
734 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
735 ++p; ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
736 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
737 }
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
738 }
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
739
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
740 uint c = *p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
741 assert(end - p != 0);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
742 switch (end - p)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
743 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
744 case 1:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
745 goto L1character;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
746 case 2:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
747 c <<= 8; c |= p[1];
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
748 goto L2characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
749 case 3:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
750 c <<= 8; c |= p[1]; c <<= 8; c |= p[2];
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
751 goto L3characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
752 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
753 version(BigEndian)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
754 c = *cast(uint*)p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
755 else
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
756 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
757 c <<= 8; c |= p[1]; c <<= 8; c |= p[2]; c <<= 8; c |= p[3];
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
758 /+
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
759 c = *cast(uint*)p;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
760 asm
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
761 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
762 mov EDX, c;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
763 bswap EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
764 mov c, EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
765 }
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
766 +/
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
767 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
768 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
769
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
770 // 4 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
771 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
772 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
773 case toUint!(">>>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
774 t.kind = TOK.RShiftAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
775 goto Lcommon_4;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
776 case toUint!("!<>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
777 t.kind = TOK.Unordered;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
778 Lcommon_4:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
779 p += 4;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
780 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
781 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
782 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
783 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
784
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
785 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
786 L3characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
787 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
788 // 3 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
789 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
790 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
791 case toUint!(">>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
792 t.kind = TOK.RShiftAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
793 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
794 case toUint!(">>>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
795 t.kind = TOK.URShift;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
796 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
797 case toUint!("<>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
798 t.kind = TOK.LorEorG;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
799 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
800 case toUint!("<<="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
801 t.kind = TOK.LShiftAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
802 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
803 case toUint!("!<="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
804 t.kind = TOK.UorG;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
805 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
806 case toUint!("!>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
807 t.kind = TOK.UorL;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
808 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
809 case toUint!("!<>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
810 t.kind = TOK.UorE;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
811 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
812 case toUint!("..."):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
813 t.kind = TOK.Ellipses;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
814 Lcommon_3:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
815 p += 3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
816 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
817 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
818 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
819 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
820
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
821 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
822 L2characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
823 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
824 // 2 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
825 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
826 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
827 case toUint!("/+"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
828 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
829 return scanNestedComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
830 case toUint!("/*"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
831 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
832 return scanBlockComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
833 case toUint!("//"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
834 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
835 assert(*p == '/');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
836 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
837 isascii(*p) || decodeUTF8();
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
838 t.kind = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
839 t.setWhitespaceFlag();
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
840 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
841 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
842 case toUint!(">="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
843 t.kind = TOK.GreaterEqual;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
844 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
845 case toUint!(">>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
846 t.kind = TOK.RShift;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
847 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
848 case toUint!("<<"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
849 t.kind = TOK.LShift;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
850 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
851 case toUint!("<="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
852 t.kind = TOK.LessEqual;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
853 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
854 case toUint!("<>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
855 t.kind = TOK.LorG;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
856 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
857 case toUint!("!<"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
858 t.kind = TOK.UorGorE;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
859 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
860 case toUint!("!>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
861 t.kind = TOK.UorLorE;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
862 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
863 case toUint!("!="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
864 t.kind = TOK.NotEqual;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
865 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
866 case toUint!(".."):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
867 t.kind = TOK.Slice;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
868 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
869 case toUint!("&&"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
870 t.kind = TOK.AndLogical;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
871 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
872 case toUint!("&="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
873 t.kind = TOK.AndAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
874 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
875 case toUint!("||"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
876 t.kind = TOK.OrLogical;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
877 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
878 case toUint!("|="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
879 t.kind = TOK.OrAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
880 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
881 case toUint!("++"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
882 t.kind = TOK.PlusPlus;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
883 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
884 case toUint!("+="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
885 t.kind = TOK.PlusAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
886 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
887 case toUint!("--"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
888 t.kind = TOK.MinusMinus;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
889 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
890 case toUint!("-="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
891 t.kind = TOK.MinusAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
892 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
893 case toUint!("=="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
894 t.kind = TOK.Equal;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
895 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
896 case toUint!("~="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
897 t.kind = TOK.CatAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
898 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
899 case toUint!("*="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
900 t.kind = TOK.MulAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
901 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
902 case toUint!("/="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
903 t.kind = TOK.DivAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
904 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
905 case toUint!("^="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
906 t.kind = TOK.XorAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
907 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
908 case toUint!("%="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
909 t.kind = TOK.ModAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
910 Lcommon_2:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
911 p += 2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
912 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
913 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
914 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
915 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
916
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
917 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
918 L1character:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
919 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
920 assert(*p == c, Format("p={0},c={1}", *p, cast(dchar)c));
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
921 // 1 character tokens.
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
922 // TODO: consider storing the token type in ptable.
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
923 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
924 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
925 case '\'':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
926 return scanCharacterLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
927 case '`':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
928 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
929 case '"':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
930 return scanNormalStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
931 case '\\':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
932 char[] buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
933 do
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
934 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
935 c = scanEscapeSequence();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
936 if (isascii(c))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
937 buffer ~= c;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
938 else
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
939 encodeUTF8(buffer, c);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
940 } while (*p == '\\')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
941 buffer ~= 0;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
942 t.kind = TOK.String;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
943 t.str = buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
944 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
945 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
946 case '<':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
947 t.kind = TOK.Greater;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
948 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
949 case '>':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
950 t.kind = TOK.Less;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
951 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
952 case '^':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
953 t.kind = TOK.Xor;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
954 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
955 case '!':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
956 t.kind = TOK.Not;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
957 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
958 case '.':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
959 if (isdigit(p[1]))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
960 return scanReal(t);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
961 t.kind = TOK.Dot;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
962 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
963 case '&':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
964 t.kind = TOK.AndBinary;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
965 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
966 case '|':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
967 t.kind = TOK.OrBinary;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
968 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
969 case '+':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
970 t.kind = TOK.Plus;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
971 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
972 case '-':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
973 t.kind = TOK.Minus;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
974 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
975 case '=':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
976 t.kind = TOK.Assign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
977 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
978 case '~':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
979 t.kind = TOK.Tilde;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
980 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
981 case '*':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
982 t.kind = TOK.Mul;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
983 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
984 case '/':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
985 t.kind = TOK.Div;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
986 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
987 case '%':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
988 t.kind = TOK.Mod;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
989 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
990 case '(':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
991 t.kind = TOK.LParen;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
992 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
993 case ')':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
994 t.kind = TOK.RParen;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
995 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
996 case '[':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
997 t.kind = TOK.LBracket;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
998 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
999 case ']':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1000 t.kind = TOK.RBracket;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1001 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1002 case '{':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1003 t.kind = TOK.LBrace;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1004 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1005 case '}':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1006 t.kind = TOK.RBrace;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1007 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1008 case ':':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1009 t.kind = TOK.Colon;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1010 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1011 case ';':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1012 t.kind = TOK.Semicolon;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1013 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1014 case '?':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1015 t.kind = TOK.Question;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1016 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1017 case ',':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1018 t.kind = TOK.Comma;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1019 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1020 case '$':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1021 t.kind = TOK.Dollar;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1022 Lcommon:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1023 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1024 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1025 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1026 case '#':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1027 return scanSpecialTokenSequence(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1028 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1029 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1030
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1031 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1032 assert(*p == c);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1033
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1034 // TODO: consider moving isidbeg() and isdigit() up.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1035 if (isidbeg(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1036 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1037 if (c == 'r' && p[1] == '"' && ++p)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1038 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1039 if (c == 'x' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1040 return scanHexStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1041 version(D2)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1042 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1043 if (c == 'q' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1044 return scanDelimitedStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1045 if (c == 'q' && p[1] == '{')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1046 return scanTokenStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1047 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1048 // Scan identifier.
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1049 Lidentifier:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1050 do
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1051 { c = *++p; }
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
1052 while (isident(c) || !isascii(c) && isUnicodeAlpha())
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1053
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1054 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1055
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
1056 auto id = IdTable.lookup(t.srcText);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1057 t.kind = id.type;
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
1058 t.ident = id;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1059
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1060 if (t.kind == TOK.Identifier || t.isKeyword)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1061 return;
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1062 else if (t.isSpecialToken)
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1063 finalizeSpecialToken(t);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1064 else if (t.kind == TOK.EOF)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1065 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1066 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1067 assert(t.srcText == "__EOF__");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1068 }
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1069 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1070 assert(0, "unexpected token type: " ~ Token.toString(t.kind));
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1071 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1072 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1073
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1074 if (isdigit(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1075 return scanNumber(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1076
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1077 // Check for EOF
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1078 if (isEOF(c))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1079 {
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1080 assert(isEOF(*p), *p~"");
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1081 t.kind = TOK.EOF;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1082 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1083 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1084 assert(t.start == t.end);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1085 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1086 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1087
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1088 if (!isascii(c))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1089 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1090 c = decodeUTF8();
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1091 if (isUniAlpha(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1092 goto Lidentifier;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1093 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1094
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1095 error(t.start, MID.IllegalCharacter, cast(dchar)c);
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1096
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1097 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1098 t.kind = TOK.Illegal;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
1099 t.setWhitespaceFlag();
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1100 t.dchar_ = c;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1101 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1102 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1103 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1104
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1105 void scanBlockComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1106 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1107 assert(p[-1] == '/' && *p == '*');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1108 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1109 auto tokenLineBegin = lineBegin;
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1110 Loop:
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1111 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1112 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1113 switch (*++p)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1114 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1115 case '*':
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1116 if (p[1] != '/')
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1117 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1118 p += 2;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1119 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1120 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1121 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1122 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1123 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1124 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1125 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1126 setLineBegin(p+1);
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1127 break;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1128 default:
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1129 if (!isascii(*p))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1130 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1131 if (isUnicodeNewlineChar(decodeUTF8()))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1132 goto case '\n';
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1133 }
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1134 else if (isEOF(*p))
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1135 {
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1136 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedBlockComment);
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1137 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1138 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1139 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1140 }
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1141 t.kind = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
1142 t.setWhitespaceFlag();
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1143 t.end = p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1144 return;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1145 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1146
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1147 void scanNestedComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1148 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1149 assert(p[-1] == '/' && *p == '+');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1150 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1151 auto tokenLineBegin = lineBegin;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1152 uint level = 1;
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1153 Loop:
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1154 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1155 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1156 switch (*++p)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1157 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1158 case '/':
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1159 if (p[1] == '+')
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1160 ++p, ++level;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1161 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1162 case '+':
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1163 if (p[1] != '/')
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1164 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1165 ++p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1166 if (--level != 0)
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1167 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1168 ++p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1169 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1170 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1171 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1172 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1173 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1174 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1175 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1176 setLineBegin(p+1);
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1177 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1178 default:
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1179 if (!isascii(*p))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1180 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1181 if (isUnicodeNewlineChar(decodeUTF8()))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1182 goto case '\n';
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1183 }
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1184 else if (isEOF(*p))
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1185 {
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1186 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedNestedComment);
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1187 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1188 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1189 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1190 }
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1191 t.kind = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
1192 t.setWhitespaceFlag();
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1193 t.end = p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1194 return;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1195 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1196
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1197 char scanPostfix()
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1198 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1199 assert(p[-1] == '"' || p[-1] == '`' ||
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1200 { version(D2) return p[-1] == '}';
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1201 else return 0; }()
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1202 );
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1203 switch (*p)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1204 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1205 case 'c':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1206 case 'w':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1207 case 'd':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1208 return *p++;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1209 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1210 return 0;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1211 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1212 assert(0);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1213 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1214
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1215 void scanNormalStringLiteral(ref Token t)
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1216 {
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1217 assert(*p == '"');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1218 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1219 auto tokenLineBegin = lineBegin;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1220 t.kind = TOK.String;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1221 char[] buffer;
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1222 uint c;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1223 while (1)
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1224 {
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1225 c = *++p;
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1226 switch (c)
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1227 {
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1228 case '"':
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1229 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1230 t.pf = scanPostfix();
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1231 Lreturn:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1232 t.str = buffer ~ '\0';
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1233 t.end = p;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1234 return;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1235 case '\\':
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1236 c = scanEscapeSequence();
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1237 --p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1238 if (isascii(c))
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1239 break;
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1240 encodeUTF8(buffer, c);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1241 continue;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1242 case '\r':
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1243 if (p[1] == '\n')
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1244 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1245 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1246 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1247 c = '\n'; // Convert Newline to \n.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1248 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1249 setLineBegin(p+1);
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1250 break;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1251 case 0, _Z_:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1252 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedString);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1253 goto Lreturn;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1254 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1255 if (!isascii(c))
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1256 {
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1257 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1258 if (isUnicodeNewlineChar(c))
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1259 goto case '\n';
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1260 encodeUTF8(buffer, c);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1261 continue;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1262 }
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1263 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1264 assert(isascii(c));
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1265 buffer ~= c;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1266 }
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
1267 assert(0);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1268 }
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1269
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1270 void scanCharacterLiteral(ref Token t)
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1271 {
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1272 assert(*p == '\'');
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1273 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1274 t.kind = TOK.CharLiteral;
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1275 switch (*p)
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1276 {
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1277 case '\\':
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1278 t.dchar_ = scanEscapeSequence();
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1279 break;
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1280 case '\'':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1281 error(t.start, MID.EmptyCharacterLiteral);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1282 break;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1283 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1284 if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1285 break;
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1286 uint c = *p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1287 if (!isascii(c))
49
7f0fa15dcffc - Renamed function.
aziz
parents: 48
diff changeset
1288 c = decodeUTF8();
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1289 t.dchar_ = c;
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1290 ++p;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1291 }
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1292
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1293 if (*p == '\'')
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1294 ++p;
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1295 else
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1296 error(t.start, MID.UnterminatedCharacterLiteral);
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1297 t.end = p;
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1298 }
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1299
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1300 void scanRawStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1301 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1302 assert(*p == '`' || *p == '"' && p[-1] == 'r');
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1303 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1304 auto tokenLineBegin = lineBegin;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1305 t.kind = TOK.String;
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1306 uint delim = *p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1307 char[] buffer;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1308 uint c;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1309 while (1)
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1310 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1311 c = *++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1312 switch (c)
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1313 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1314 case '\r':
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1315 if (p[1] == '\n')
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1316 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1317 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1318 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1319 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1320 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1321 setLineBegin(p+1);
52
f65a83c27638 - Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well.
aziz
parents: 51
diff changeset
1322 break;
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1323 case '`':
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1324 case '"':
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1325 if (c == delim)
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1326 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1327 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1328 t.pf = scanPostfix();
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1329 Lreturn:
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1330 t.str = buffer ~ '\0';
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1331 t.end = p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1332 return;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1333 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1334 break;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1335 case 0, _Z_:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1336 error(tokenLineNum, tokenLineBegin, t.start,
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1337 delim == 'r' ? MID.UnterminatedRawString : MID.UnterminatedBackQuoteString);
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1338 goto Lreturn;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1339 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1340 if (!isascii(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1341 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1342 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1343 if (isUnicodeNewlineChar(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1344 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1345 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1346 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1347 }
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1348 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1349 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1350 buffer ~= c;
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1351 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1352 assert(0);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1353 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1354
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1355 void scanHexStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1356 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1357 assert(p[0] == 'x' && p[1] == '"');
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1358 t.kind = TOK.String;
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1359
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1360 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1361 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1362
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1363 uint c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1364 ubyte[] buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1365 ubyte h; // hex number
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1366 uint n; // number of hex digits
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1367
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1368 ++p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1369 assert(*p == '"');
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1370 while (1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1371 {
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1372 c = *++p;
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1373 switch (c)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1374 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1375 case '"':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1376 if (n & 1)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1377 error(tokenLineNum, tokenLineBegin, t.start, MID.OddNumberOfDigitsInHexString);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1378 ++p;
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1379 t.pf = scanPostfix();
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1380 Lreturn:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1381 t.str = cast(string) (buffer ~= 0);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1382 t.end = p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1383 return;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1384 case '\r':
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1385 if (p[1] == '\n')
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1386 ++p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1387 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1388 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1389 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1390 setLineBegin(p+1);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1391 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1392 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1393 if (ishexad(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1394 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1395 if (c <= '9')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1396 c -= '0';
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1397 else if (c <= 'F')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1398 c -= 'A' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1399 else
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1400 c -= 'a' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1401
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1402 if (n & 1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1403 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1404 h <<= 4;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1405 h |= c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1406 buffer ~= h;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1407 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1408 else
112
004d98df65af - Implemented parseInterfaceDeclaration().
aziz
parents: 105
diff changeset
1409 h = cast(ubyte)c;
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1410 ++n;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1411 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1412 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1413 else if (isspace(c))
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
1414 continue; // Skip spaces.
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
1415 else if (isEOF(c))
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1416 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1417 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedHexString);
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1418 t.pf = 0;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1419 goto Lreturn;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1420 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1421 else
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1422 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1423 auto errorAt = p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1424 if (!isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1425 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1426 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1427 if (isUnicodeNewlineChar(c))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1428 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1429 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1430 error(errorAt, MID.NonHexCharInHexString, cast(dchar)c);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1431 }
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1432 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1433 }
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1434 assert(0);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1435 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1436
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1437 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1438 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1439 void scanDelimitedStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1440 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1441 assert(p[0] == 'q' && p[1] == '"');
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1442 t.kind = TOK.String;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1443
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1444 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1445 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1446
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1447 char[] buffer;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1448 dchar opening_delim = 0, // 0 if no nested delimiter or '[', '(', '<', '{'
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1449 closing_delim; // Will be ']', ')', '>', '},
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1450 // the first character of an identifier or
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1451 // any other Unicode/ASCII character.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1452 char[] str_delim; // Identifier delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1453 uint level = 1; // Counter for nestable delimiters.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1454
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1455 ++p; ++p; // Skip q"
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1456 uint c = *p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1457 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1458 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1459 case '(':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1460 opening_delim = c;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1461 closing_delim = ')'; // c + 1
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1462 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1463 case '[', '<', '{':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1464 opening_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1465 closing_delim = c + 2; // Get to closing counterpart. Feature of ASCII table.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1466 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1467 default:
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1468 dchar scanNewline()
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1469 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1470 switch (*p)
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1471 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1472 case '\r':
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1473 if (p[1] == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1474 ++p;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1475 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1476 assert(isNewlineEnd(p));
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1477 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1478 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1479 setLineBegin(p);
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1480 return '\n';
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1481 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1482 if (isUnicodeNewline(p))
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1483 {
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
1484 ++p; ++p;
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
1485 goto case '\n';
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1486 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1487 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1488 return 0;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1489 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1490 // Skip leading newlines:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1491 while (scanNewline() != 0)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1492 {}
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1493 assert(!isNewline(p));
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1494
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1495 char* begin = p;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1496 c = *p;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1497 closing_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1498 // TODO: Check for non-printable characters?
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1499 if (!isascii(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1500 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1501 closing_delim = decodeUTF8();
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1502 if (!isUniAlpha(closing_delim))
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1503 break; // Not an identifier.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1504 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1505 else if (!isidbeg(c))
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1506 break; // Not an identifier.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1507
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1508 // Parse Identifier + EndOfLine
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1509 do
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1510 { c = *++p; }
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
1511 while (isident(c) || !isascii(c) && isUnicodeAlpha())
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1512 // Store identifier
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1513 str_delim = begin[0..p-begin];
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1514 // Scan newline
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1515 if (scanNewline() == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1516 --p; // Go back one because of "c = *++p;" in main loop.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1517 else
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1518 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1519 // TODO: error(p, MID.ExpectedNewlineAfterIdentDelim);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1520 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1521 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1522
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1523 bool checkStringDelim(char* p)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1524 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1525 assert(str_delim.length != 0);
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1526 if (buffer[$-1] == '\n' && // Last character copied to buffer must be '\n'.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1527 end-p >= str_delim.length && // Check remaining length.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1528 p[0..str_delim.length] == str_delim) // Compare.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1529 return true;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1530 return false;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1531 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1532
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1533 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1534 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1535 c = *++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1536 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1537 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1538 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1539 if (p[1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1540 ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1541 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1542 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1543 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1544 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1545 setLineBegin(p+1);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1546 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1547 case 0, _Z_:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1548 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedDelimitedString);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1549 goto Lreturn3;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1550 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1551 if (!isascii(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1552 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1553 auto begin = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1554 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1555 if (isUnicodeNewlineChar(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1556 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1557 if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1558 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1559 if (str_delim.length)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1560 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1561 if (checkStringDelim(begin))
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1562 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1563 p = begin + str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1564 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1565 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1566 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1567 else
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1568 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1569 assert(level == 1);
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1570 --level;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1571 goto Lreturn;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1572 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1573 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1574 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1575 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1576 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1577 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1578 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1579 if (c == opening_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1580 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1581 else if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1582 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1583 if (str_delim.length)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1584 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1585 if (checkStringDelim(p))
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1586 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1587 p += str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1588 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1589 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1590 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1591 else if (--level == 0)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1592 goto Lreturn;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1593 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1594 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1595 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1596 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1597 buffer ~= c;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1598 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1599 Lreturn: // Character delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1600 assert(c == closing_delim);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1601 assert(level == 0);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1602 ++p; // Skip closing delimiter.
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1603 Lreturn2: // String delimiter.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1604 if (*p == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1605 ++p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1606 else
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1607 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1608 // TODO: error(p, MID.ExpectedDblQuoteAfterDelim, str_delim.length ? str_delim : closing_delim~"");
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1609 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1610
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1611 t.pf = scanPostfix();
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1612 Lreturn3: // Error.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1613 t.str = buffer ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1614 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1615 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1616
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1617 void scanTokenStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1618 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1619 assert(p[0] == 'q' && p[1] == '{');
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1620 t.kind = TOK.String;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1621
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1622 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1623 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1624
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1625 // A guard against changes to particular members:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1626 // this.lineNum_hline and this.errorPath
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1627 ++inTokenString;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1628
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1629 uint lineNum = this.lineNum;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1630 uint level = 1;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1631
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1632 ++p; ++p; // Skip q{
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1633
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1634 auto prev_t = &t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1635 Token* token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1636 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1637 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1638 token = new Token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1639 scan(*token);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1640 // Save the tokens in a doubly linked list.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1641 // Could be useful for various tools.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1642 token.prev = prev_t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1643 prev_t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1644 prev_t = token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1645 switch (token.kind)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1646 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1647 case TOK.LBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1648 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1649 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1650 case TOK.RBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1651 if (--level == 0)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1652 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1653 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1654 t.next = null;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1655 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1656 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1657 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1658 case TOK.EOF:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1659 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedTokenString);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1660 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1661 t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1662 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1663 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1664 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1665 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1666 break; // Exit loop.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1667 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1668
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1669 assert(token.kind == TOK.RBrace || token.kind == TOK.EOF);
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1670 assert(token.kind == TOK.RBrace && t.next is null ||
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1671 token.kind == TOK.EOF && t.next !is null);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1672
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1673 char[] buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1674 // token points to } or EOF
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1675 if (token.kind == TOK.EOF)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1676 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1677 t.end = token.start;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1678 buffer = t.srcText[2..$].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1679 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1680 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1681 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1682 // Assign to buffer before scanPostfix().
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1683 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1684 buffer = t.srcText[2..$-1].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1685 t.pf = scanPostfix();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1686 t.end = p; // Assign again because of postfix.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1687 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1688 // Convert newlines to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1689 if (lineNum != this.lineNum)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1690 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1691 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1692 uint i, j;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1693 for (; i < buffer.length; ++i)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1694 switch (buffer[i])
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1695 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1696 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1697 if (buffer[i+1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1698 ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1699 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1700 assert(isNewlineEnd(buffer.ptr + i));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1701 buffer[j++] = '\n'; // Convert Newline to '\n'.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1702 break;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1703 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1704 if (isUnicodeNewline(buffer.ptr + i))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1705 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1706 ++i; ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1707 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1708 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1709 buffer[j++] = buffer[i]; // Copy.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1710 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1711 buffer.length = j; // Adjust length.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1712 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1713 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1714 t.str = buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1715
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1716 --inTokenString;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1717 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1718 } // version(D2)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1719
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1720 dchar scanEscapeSequence()
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1721 out(result)
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
1722 { assert(isValidChar(result)); }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1723 body
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1724 {
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1725 assert(*p == '\\');
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1726
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1727 auto sequenceStart = p; // Used for error reporting.
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1728
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1729 ++p;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1730 uint c = char2ev(*p);
82
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1731 if (c)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1732 {
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1733 ++p;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1734 return c;
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1735 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1736
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1737 uint digits = 2;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1738
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1739 switch (*p)
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1740 {
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1741 case 'x':
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1742 assert(c == 0);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1743 while (1)
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1744 {
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1745 ++p;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1746 if (ishexad(*p))
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1747 {
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1748 c *= 16;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1749 if (*p <= '9')
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1750 c += *p - '0';
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1751 else if (*p <= 'F')
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1752 c += *p - 'A' + 10;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1753 else
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1754 c += *p - 'a' + 10;
82
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1755
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1756 if (!--digits)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1757 {
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1758 ++p;
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
1759 if (isValidChar(c))
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1760 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1761
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1762 error(sequenceStart, MID.InvalidUnicodeEscapeSequence, sequenceStart[0..p-sequenceStart]);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1763 break;
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1764 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1765 continue;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1766 }
432
3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 430
diff changeset
1767
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1768 error(sequenceStart, MID.InsufficientHexDigits);
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1769 break;
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1770 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1771 break;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1772 case 'u':
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1773 digits = 4;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1774 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1775 case 'U':
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1776 digits = 8;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1777 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1778 default:
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1779 if (isoctal(*p))
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1780 {
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1781 assert(c == 0);
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1782 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1783 ++p;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1784 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1785 return c;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1786 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1787 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1788 ++p;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1789 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1790 return c;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1791 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1792 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1793 ++p;
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1794 return c; // Return valid escape value.
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1795 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1796 else if(*p == '&')
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1797 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1798 if (isalpha(*++p))
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1799 {
272
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1800 auto begin = p;
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1801 while (isalnum(*++p))
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1802 {}
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1803
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1804 if (*p == ';')
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1805 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1806 // Pass entity excluding '&' and ';'.
272
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1807 c = entity2Unicode(begin[0..p - begin]);
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1808 ++p; // Skip ;
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1809 if (c != 0xFFFF)
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1810 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1811 else
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1812 error(sequenceStart, MID.UndefinedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
272
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1813 }
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1814 else
432
3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 430
diff changeset
1815 error(sequenceStart, MID.UnterminatedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1816 }
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1817 else
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1818 error(sequenceStart, MID.InvalidBeginHTMLEntity);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1819 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1820 else if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1821 error(sequenceStart, MID.UndefinedEscapeSequence,
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
1822 isEOF(*p) ? `\EOF` : `\NewLine`);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1823 else
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1824 {
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1825 char[] str = `\`;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1826 if (isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1827 str ~= *p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1828 else
387
ad0cbd1c8881 Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 386
diff changeset
1829 encodeUTF8(str, decodeUTF8());
ad0cbd1c8881 Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 386
diff changeset
1830 ++p;
ad0cbd1c8881 Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 386
diff changeset
1831 // TODO: check for unprintable character?
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1832 error(sequenceStart, MID.UndefinedEscapeSequence, str);
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1833 }
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1834 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1835 return REPLACEMENT_CHAR; // Error: return replacement character.
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1836 }
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1837
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1838 /*
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1839 IntegerLiteral:= (Dec|Hex|Bin|Oct)Suffix?
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1840 Dec:= (0|[1-9][0-9_]*)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1841 Hex:= 0[xX] HexDigits
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1842 Bin:= 0[bB][01_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1843 Oct:= 0[0-7_]+
68
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
1844 Suffix:= (L[uU]?|[uU]L?)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1845 HexDigits:= [0-9a-zA-Z_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1846
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1847 Invalid: "0b_", "0x_", "._"
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1848 */
15
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
1849 void scanNumber(ref Token t)
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
1850 {
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1851 ulong ulong_;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1852 bool overflow;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1853 bool isDecimal;
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1854 size_t digits;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1855
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1856 if (*p != '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1857 goto LscanInteger;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1858 ++p; // skip zero
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1859 // check for xX bB ...
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1860 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1861 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1862 case 'x','X':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1863 goto LscanHex;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1864 case 'b','B':
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1865 goto LscanBinary;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1866 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1867 if (p[1] == 'i')
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1868 goto LscanReal; // 0Li
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1869 break; // 0L
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1870 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1871 if (p[1] == '.')
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1872 break; // 0..
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1873 // 0.
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1874 case 'i','f','F', // Imaginary and float literal suffixes.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1875 'e', 'E': // Float exponent.
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1876 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1877 default:
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1878 if (*p == '_')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1879 goto LscanOctal; // 0_
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1880 else if (isdigit(*p))
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1881 {
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1882 if (*p == '8' || *p == '9')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1883 goto Loctal_hasDecimalDigits; // 08 or 09
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1884 else
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1885 goto Loctal_enter_loop; // 0[0-7]
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1886 }
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1887 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1888
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1889 // Number 0
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1890 assert(p[-1] == '0');
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1891 assert(*p != '_' && !isdigit(*p));
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1892 assert(ulong_ == 0);
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1893 isDecimal = true;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1894 goto Lfinalize;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1895
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1896 LscanInteger:
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1897 assert(*p != 0 && isdigit(*p));
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1898 isDecimal = true;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1899 goto Lenter_loop_int;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1900 while (1)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1901 {
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1902 if (*++p == '_')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1903 continue;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1904 if (!isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1905 break;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1906 Lenter_loop_int:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1907 if (ulong_ < ulong.max/10 || (ulong_ == ulong.max/10 && *p <= '5'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1908 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1909 ulong_ *= 10;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1910 ulong_ += *p - '0';
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1911 continue;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1912 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1913 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1914 overflow = true;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1915 while (isdigit(*++p)) {}
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1916 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1917 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1918
61
512cd2248dfc - Fix: issueing error on hexadecimal number overflow.
aziz
parents: 60
diff changeset
1919 // The number could be a float, so check overflow below.
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1920 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1921 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1922 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1923 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1924 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1925 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1926 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1927 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1928 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1929 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1930 goto LscanReal;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1931 default:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1932 }
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1933
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1934 if (overflow)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1935 error(t.start, MID.OverflowDecimalNumber);
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1936
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1937 assert((isdigit(p[-1]) || p[-1] == '_') && !isdigit(*p) && *p != '_');
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1938 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1939
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1940 LscanHex:
61
512cd2248dfc - Fix: issueing error on hexadecimal number overflow.
aziz
parents: 60
diff changeset
1941 assert(digits == 0);
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
1942 assert(*p == 'x' || *p == 'X');
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1943 while (1)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1944 {
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1945 if (*++p == '_')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1946 continue;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1947 if (!ishexad(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1948 break;
61
512cd2248dfc - Fix: issueing error on hexadecimal number overflow.
aziz
parents: 60
diff changeset
1949 ++digits;
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1950 ulong_ *= 16;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1951 if (*p <= '9')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1952 ulong_ += *p - '0';
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1953 else if (*p <= 'F')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1954 ulong_ += *p - 'A' + 10;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1955 else
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1956 ulong_ += *p - 'a' + 10;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1957 }
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1958
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
1959 assert(ishexad(p[-1]) || p[-1] == '_' || p[-1] == 'x' || p[-1] == 'X');
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1960 assert(!ishexad(*p) && *p != '_');
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1961
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1962 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1963 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1964 case '.':
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1965 if (p[1] == '.')
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1966 break;
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1967 case 'p', 'P':
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1968 return scanHexReal(t);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1969 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1970 }
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1971
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1972 if (digits == 0 || digits > 16)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1973 error(t.start, digits == 0 ? MID.NoDigitsInHexNumber : MID.OverflowHexNumber);
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1974
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1975 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1976
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1977 LscanBinary:
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1978 assert(digits == 0);
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
1979 assert(*p == 'b' || *p == 'B');
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1980 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1981 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1982 if (*++p == '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1983 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1984 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1985 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1986 }
367
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1987 else if (*p == '1')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1988 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1989 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1990 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1991 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1992 }
367
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1993 else if (*p == '_')
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1994 continue;
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1995 else
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1996 break;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1997 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1998
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1999 if (digits == 0 || digits > 64)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2000 error(t.start, digits == 0 ? MID.NoDigitsInBinNumber : MID.OverflowBinaryNumber);
59
3e594725899a - Issuing error when no digits were found in hex and binary numbers.
aziz
parents: 58
diff changeset
2001
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
2002 assert(p[-1] == '0' || p[-1] == '1' || p[-1] == '_' || p[-1] == 'b' || p[-1] == 'B', p[-1] ~ "");
367
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
2003 assert( !(*p == '0' || *p == '1' || *p == '_') );
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2004 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2005
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2006 LscanOctal:
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2007 assert(*p == '_');
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2008 while (1)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2009 {
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2010 if (*++p == '_')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2011 continue;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2012 if (!isoctal(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2013 break;
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2014 Loctal_enter_loop:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2015 if (ulong_ < ulong.max/2 || (ulong_ == ulong.max/2 && *p <= '1'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2016 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2017 ulong_ *= 8;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2018 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2019 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2020 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2021 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2022 overflow = true;
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2023 while (isoctal(*++p)) {}
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2024 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2025 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2026
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2027 bool hasDecimalDigits;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2028 if (isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2029 {
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2030 Loctal_hasDecimalDigits:
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2031 hasDecimalDigits = true;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2032 while (isdigit(*++p)) {}
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2033 }
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2034
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2035 // The number could be a float, so check errors below.
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2036 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2037 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2038 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2039 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2040 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2041 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2042 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2043 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2044 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2045 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2046 goto LscanReal;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2047 default:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2048 }
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2049
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2050 if (hasDecimalDigits)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2051 error(t.start, MID.OctalNumberHasDecimals);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2052
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2053 if (overflow)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2054 error(t.start, MID.OverflowOctalNumber);
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2055 // goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2056
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2057 Lfinalize:
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2058 enum Suffix
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2059 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2060 None = 0,
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2061 Unsigned = 1,
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2062 Long = 2
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2063 }
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2064
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2065 // Scan optional suffix: L, Lu, LU, u, uL, U or UL.
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2066 Suffix suffix;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2067 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2068 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2069 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2070 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2071 case 'L':
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2072 if (suffix & Suffix.Long)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2073 break;
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2074 suffix |= Suffix.Long;
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2075 ++p;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2076 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2077 case 'u', 'U':
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2078 if (suffix & Suffix.Unsigned)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2079 break;
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2080 suffix |= Suffix.Unsigned;
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2081 ++p;
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2082 continue;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2083 default:
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2084 break;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2085 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2086 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2087 }
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2088
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2089 // Determine type of Integer.
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2090 switch (suffix)
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2091 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2092 case Suffix.None:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2093 if (ulong_ & 0x8000_0000_0000_0000)
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2094 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2095 if (isDecimal)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2096 error(t.start, MID.OverflowDecimalSign);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2097 t.kind = TOK.Uint64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2098 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2099 else if (ulong_ & 0xFFFF_FFFF_0000_0000)
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2100 t.kind = TOK.Int64;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2101 else if (ulong_ & 0x8000_0000)
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2102 t.kind = isDecimal ? TOK.Int64 : TOK.Uint32;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2103 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2104 t.kind = TOK.Int32;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2105 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2106 case Suffix.Unsigned:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2107 if (ulong_ & 0xFFFF_FFFF_0000_0000)
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2108 t.kind = TOK.Uint64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2109 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2110 t.kind = TOK.Uint32;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2111 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2112 case Suffix.Long:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2113 if (ulong_ & 0x8000_0000_0000_0000)
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2114 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2115 if (isDecimal)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2116 error(t.start, MID.OverflowDecimalSign);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2117 t.kind = TOK.Uint64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2118 }
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2119 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2120 t.kind = TOK.Int64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2121 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2122 case Suffix.Unsigned | Suffix.Long:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2123 t.kind = TOK.Uint64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2124 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2125 default:
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2126 assert(0);
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2127 }
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2128 t.ulong_ = ulong_;
15
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
2129 t.end = p;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2130 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2131 LscanReal:
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2132 scanReal(t);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2133 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2134 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2135
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2136 /*
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2137 FloatLiteral:= Float[fFL]?i?
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2138 Float:= DecFloat | HexFloat
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2139 DecFloat:= ([0-9][0-9_]*[.][0-9_]*DecExponent?) | [.][0-9][0-9_]*DecExponent? | [0-9][0-9_]*DecExponent
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2140 DecExponent:= [eE][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2141 HexFloat:= 0[xX](HexDigits[.]HexDigits | [.][0-9a-zA-Z]HexDigits? | HexDigits)HexExponent
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2142 HexExponent:= [pP][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2143 */
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2144 void scanReal(ref Token t)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2145 {
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2146 if (*p == '.')
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2147 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2148 assert(p[1] != '.');
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2149 // This function was called by scan() or scanNumber().
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2150 while (isdigit(*++p) || *p == '_') {}
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2151 }
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2152 else
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2153 // This function was called by scanNumber().
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2154 assert(delegate ()
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2155 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2156 switch (*p)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2157 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2158 case 'L':
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2159 if (p[1] != 'i')
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2160 return false;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2161 case 'i', 'f', 'F', 'e', 'E':
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2162 return true;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2163 default:
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2164 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2165 return false;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2166 }()
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2167 );
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2168
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2169 // Scan exponent.
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2170 if (*p == 'e' || *p == 'E')
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2171 {
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2172 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2173 if (*p == '-' || *p == '+')
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2174 ++p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2175 if (isdigit(*p))
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2176 while (isdigit(*++p) || *p == '_') {}
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2177 else
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2178 error(t.start, MID.FloatExpMustStartWithDigit);
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2179 }
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2180
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2181 // Copy whole number and remove underscores from buffer.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2182 char[] buffer = t.start[0..p-t.start].dup;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2183 uint j;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2184 foreach (c; buffer)
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2185 if (c != '_')
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2186 buffer[j++] = c;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2187 buffer.length = j; // Adjust length.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2188 buffer ~= 0; // Terminate for C functions.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2189
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2190 finalizeFloat(t, buffer);
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2191 }
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2192
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2193 void scanHexReal(ref Token t)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2194 {
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2195 assert(*p == '.' || *p == 'p' || *p == 'P');
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2196 MID mid;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2197 if (*p == '.')
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2198 while (ishexad(*++p) || *p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2199 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2200 // Decimal exponent is required.
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2201 if (*p != 'p' && *p != 'P')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2202 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2203 mid = MID.HexFloatExponentRequired;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2204 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2205 }
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2206 // Scan exponent
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2207 assert(*p == 'p' || *p == 'P');
393
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2208 ++p;
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2209 if (*p == '+' || *p == '-')
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2210 ++p;
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2211 if (!isdigit(*p))
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2212 {
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2213 mid = MID.HexFloatExpMustStartWithDigit;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2214 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2215 }
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2216 while (isdigit(*++p) || *p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2217 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2218 // Copy whole number and remove underscores from buffer.
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2219 char[] buffer = t.start[0..p-t.start].dup;
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2220 uint j;
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2221 foreach (c; buffer)
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2222 if (c != '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2223 buffer[j++] = c;
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2224 buffer.length = j; // Adjust length.
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2225 buffer ~= 0; // Terminate for C functions.
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2226 finalizeFloat(t, buffer);
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2227 return;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2228 Lerr:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2229 t.kind = TOK.Float32;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2230 t.end = p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2231 error(t.start, mid);
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2232 }
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2233
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2234 void finalizeFloat(ref Token t, string buffer)
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2235 {
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2236 assert(buffer[$-1] == 0);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2237 // Float number is well-formed. Check suffixes and do conversion.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2238 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2239 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2240 case 'f', 'F':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2241 t.kind = TOK.Float32;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2242 t.float_ = strtof(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2243 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2244 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2245 case 'L':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2246 t.kind = TOK.Float80;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2247 t.real_ = strtold(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2248 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2249 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2250 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2251 t.kind = TOK.Float64;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2252 t.double_ = strtod(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2253 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2254 if (*p == 'i')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2255 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2256 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2257 t.kind += 3; // Switch to imaginary counterpart.
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2258 assert(t.kind == TOK.Imaginary32 ||
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2259 t.kind == TOK.Imaginary64 ||
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2260 t.kind == TOK.Imaginary80);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2261 }
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
2262 if (errno() == ERANGE)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2263 error(t.start, MID.OverflowFloatNumber);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2264 t.end = p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2265 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2266
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2267 /// Scan special token: #line Integer [Filespec] EndOfLine
360
b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence().
aziz
parents: 350
diff changeset
2268 void scanSpecialTokenSequence(ref Token t)
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2269 {
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
2270 assert(*p == '#');
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2271 t.kind = TOK.HashLine;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
2272 t.setWhitespaceFlag();
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2273
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2274 MID mid;
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2275 char* errorAtColumn = p;
763
f26f13b5a3a3 Fixed code in scanSpecialTokenSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 762
diff changeset
2276 char* tokenEnd = ++p;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2277
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2278 if (!(p[0] == 'l' && p[1] == 'i' && p[2] == 'n' && p[3] == 'e'))
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2279 {
370
ae4afb66768f - Renamed findModule() to findModulePath().
aziz
parents: 367
diff changeset
2280 mid = MID.ExpectedIdentifierSTLine;
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2281 goto Lerr;
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2282 }
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2283 p += 3;
763
f26f13b5a3a3 Fixed code in scanSpecialTokenSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 762
diff changeset
2284 tokenEnd = p + 1;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2285
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2286 // TODO: #line58"path/file" is legal. Require spaces?
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2287 // State.Space could be used for that purpose.
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2288 enum State
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2289 { /+Space,+/ Integer, Filespec, End }
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2290
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2291 State state = State.Integer;
763
f26f13b5a3a3 Fixed code in scanSpecialTokenSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 762
diff changeset
2292
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2293 while (!isEndOfLine(++p))
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2294 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2295 if (isspace(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2296 continue;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2297 if (state == State.Integer)
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2298 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2299 if (!isdigit(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2300 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2301 errorAtColumn = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2302 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2303 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2304 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2305 t.tokLineNum = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2306 scan(*t.tokLineNum);
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2307 tokenEnd = p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2308 if (t.tokLineNum.kind != TOK.Int32 && t.tokLineNum.kind != TOK.Uint32)
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2309 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2310 errorAtColumn = t.tokLineNum.start;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2311 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2312 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2313 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2314 --p; // Go one back because scan() advanced p past the integer.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2315 state = State.Filespec;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2316 }
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2317 else if (state == State.Filespec && *p == '"')
763
f26f13b5a3a3 Fixed code in scanSpecialTokenSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 762
diff changeset
2318 { // MID.ExpectedFilespec is deprecated.
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2319 // if (*p != '"')
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2320 // {
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2321 // errorAtColumn = p;
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2322 // mid = MID.ExpectedFilespec;
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2323 // goto Lerr;
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2324 // }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2325 t.tokLineFilespec = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2326 t.tokLineFilespec.start = p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2327 t.tokLineFilespec.kind = TOK.Filespec;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
2328 t.tokLineFilespec.setWhitespaceFlag();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2329 while (*++p != '"')
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2330 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2331 if (isEndOfLine(p))
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2332 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2333 errorAtColumn = t.tokLineFilespec.start;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2334 mid = MID.UnterminatedFilespec;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2335 t.tokLineFilespec.end = p;
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2336 tokenEnd = p;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2337 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2338 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2339 isascii(*p) || decodeUTF8();
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2340 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2341 auto start = t.tokLineFilespec.start +1; // +1 skips '"'
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2342 t.tokLineFilespec.str = start[0 .. p - start];
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2343 t.tokLineFilespec.end = p + 1;
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2344 tokenEnd = p + 1;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2345 state = State.End;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2346 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2347 else/+ if (state == State.End)+/
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2348 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2349 mid = MID.UnterminatedSpecialToken;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2350 goto Lerr;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2351 }
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2352 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2353 assert(isEndOfLine(p));
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2354
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2355 if (state == State.Integer)
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2356 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2357 errorAtColumn = p;
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2358 mid = MID.ExpectedIntegerAfterSTLine;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2359 goto Lerr;
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2360 }
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2361
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2362 // Evaluate #line only when not in token string.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2363 if (!inTokenString && t.tokLineNum)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2364 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2365 this.lineNum_hline = this.lineNum - t.tokLineNum.uint_ + 1;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2366 if (t.tokLineFilespec)
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
2367 newFilePath(t.tokLineFilespec.str);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2368 }
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2369 p = tokenEnd;
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2370 t.end = tokenEnd;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2371
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2372 return;
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2373 Lerr:
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2374 p = tokenEnd;
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2375 t.end = tokenEnd;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2376 error(errorAtColumn, mid);
371
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer.
aziz
parents: 370
diff changeset
2377 }
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer.
aziz
parents: 370
diff changeset
2378
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2379 /++
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2380 Insert an empty dummy token before t.
737
f88b5285b86b Implemented DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 715
diff changeset
2381
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2382 Useful in the parsing phase for representing a node in the AST
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2383 that doesn't consume an actual token from the source text.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2384 +/
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2385 Token* insertEmptyTokenBefore(Token* t)
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2386 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2387 assert(t !is null && t.prev !is null);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2388 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2389 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2390
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2391 auto prev_t = t.prev;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2392 auto new_t = new Token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2393 new_t.kind = TOK.Empty;
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2394 new_t.start = new_t.end = prev_t.end;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2395 // Link in new token.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2396 prev_t.next = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2397 new_t.prev = prev_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2398 new_t.next = t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2399 t.prev = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2400 return new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2401 }
105
df34ec47fb81 - Added getState() method and State struct to Lexer.
aziz
parents: 103
diff changeset
2402
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2403 uint errorLineNumber(uint lineNum)
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
2404 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2405 return lineNum - this.lineNum_hline;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
2406 }
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
2407
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2408 void error(char* columnPos, MID mid, ...)
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2409 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2410 error_(this.lineNum, this.lineBegin, columnPos, mid, _arguments, _argptr);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2411 }
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2412
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2413 void error(uint lineNum, char* lineBegin, char* columnPos, MID mid, ...)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2414 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2415 error_(lineNum, lineBegin, columnPos, mid, _arguments, _argptr);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2416 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2417
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2418 void error_(uint lineNum, char* lineBegin, char* columnPos, MID mid,
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2419 TypeInfo[] _arguments, Arg _argptr)
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2420 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2421 lineNum = this.errorLineNumber(lineNum);
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
2422 auto errorPath = this.filePaths.setPath;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2423 auto location = new Location(errorPath, lineNum, lineBegin, columnPos);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2424 auto msg = Format(_arguments, _argptr, GetMsg(mid));
520
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2425 auto error = new LexerError(location, msg);
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2426 errors ~= error;
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2427 if (infoMan !is null)
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2428 infoMan ~= error;
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2429 }
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2430
306
9c866aadcb5b - Moved code out of main() to separate functions.
aziz
parents: 272
diff changeset
2431 Token* getTokens()
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
2432 {
306
9c866aadcb5b - Moved code out of main() to separate functions.
aziz
parents: 272
diff changeset
2433 while (nextToken() != TOK.EOF)
9c866aadcb5b - Moved code out of main() to separate functions.
aziz
parents: 272
diff changeset
2434 {}
9c866aadcb5b - Moved code out of main() to separate functions.
aziz
parents: 272
diff changeset
2435 return head;
3
4bbce78bfb1e - Added TOK enum.
aziz
parents: 2
diff changeset
2436 }
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2437
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2438 /// Scan the whole text until EOF is encountered.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2439 void scanAll()
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2440 {
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2441 while (nextToken() != TOK.EOF)
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2442 {}
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2443 }
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2444
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2445 /// HEAD -> Newline -> First Token
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2446 Token* firstToken()
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2447 {
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2448 return this.head.next.next;
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2449 }
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2450
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2451 static void loadKeywords(ref Identifier[string] table)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2452 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2453 foreach(k; keywords)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2454 table[k.str] = k;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2455 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2456
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2457 /// Returns true if str is a valid D identifier.
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2458 static bool isIdentifierString(char[] str)
365
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2459 {
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2460 if (str.length == 0 || isdigit(str[0]))
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2461 return false;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2462 size_t idx;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2463 do
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2464 {
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2465 auto c = dil.Unicode.decode(str, idx);
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2466 if (c == ERROR_CHAR || !(isident(c) || !isascii(c) && isUniAlpha(c)))
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2467 return false;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2468 } while (idx < str.length)
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2469 return true;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2470 }
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2471
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2472 /// Returns true if str is a keyword or a special token (__FILE__, __LINE__ etc.)
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2473 static bool isReservedIdentifier(char[] str)
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2474 {
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2475 if (str.length == 0)
382
66477017cb95 Applied fixes to isNonReservedIdentifier().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 371
diff changeset
2476 return false;
66477017cb95 Applied fixes to isNonReservedIdentifier().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 371
diff changeset
2477
365
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2478 static Identifier[string] reserved_ids_table;
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2479 if (reserved_ids_table is null)
499
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
2480 Lexer.loadKeywords(reserved_ids_table);
365
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2481
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2482 if (!isIdentifierString(str))
365
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2483 return false;
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2484
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2485 return (str in reserved_ids_table) !is null;
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2486 }
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2487
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2488 /++
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2489 Returns true if the current character to be decoded is
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2490 a Unicode alpha character.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2491 The current pointer 'p' is not advanced if false is returned.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2492 +/
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2493 bool isUnicodeAlpha()
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2494 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2495 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2496 char* p = this.p;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2497 dchar d = *p;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2498 ++p; // Move to second byte.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2499 // Error if second byte is not a trail byte.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2500 if (!isTrailByte(*p))
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2501 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2502 // Check for overlong sequences.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2503 switch (d)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2504 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2505 case 0xE0, 0xF0, 0xF8, 0xFC:
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2506 if ((*p & d) == 0x80)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2507 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2508 default:
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2509 if ((d & 0xFE) == 0xC0) // 1100000x
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2510 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2511 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2512 const char[] checkNextByte = "if (!isTrailByte(*++p))"
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2513 " return false;";
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2514 const char[] appendSixBits = "d = (d << 6) | *p & 0b0011_1111;";
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2515 // Decode
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2516 if ((d & 0b1110_0000) == 0b1100_0000)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2517 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2518 d &= 0b0001_1111;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2519 mixin(appendSixBits);
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2520 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2521 else if ((d & 0b1111_0000) == 0b1110_0000)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2522 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2523 d &= 0b0000_1111;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2524 mixin(appendSixBits ~
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2525 checkNextByte ~ appendSixBits);
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2526 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2527 else if ((d & 0b1111_1000) == 0b1111_0000)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2528 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2529 d &= 0b0000_0111;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2530 mixin(appendSixBits ~
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2531 checkNextByte ~ appendSixBits ~
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2532 checkNextByte ~ appendSixBits);
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2533 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2534 else
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2535 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2536
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2537 assert(isTrailByte(*p));
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2538 if (!isValidChar(d) || !isUniAlpha(d))
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2539 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2540 // Only advance pointer if this is a Unicode alpha character.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2541 this.p = p;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2542 return true;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2543 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2544
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2545 /// Decodes the next UTF-8 sequence.
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2546 dchar decodeUTF8()
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2547 {
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2548 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2549 char* p = this.p;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2550 dchar d = *p;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2551
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2552 ++p; // Move to second byte.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2553 // Error if second byte is not a trail byte.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2554 if (!isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2555 goto Lerr2;
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2556
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2557 // Check for overlong sequences.
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2558 switch (d)
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2559 {
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2560 case 0xE0, // 11100000 100xxxxx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2561 0xF0, // 11110000 1000xxxx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2562 0xF8, // 11111000 10000xxx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2563 0xFC: // 11111100 100000xx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2564 if ((*p & d) == 0x80)
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2565 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2566 default:
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2567 if ((d & 0xFE) == 0xC0) // 1100000x
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2568 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2569 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2570
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2571 const char[] checkNextByte = "if (!isTrailByte(*++p))"
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2572 " goto Lerr2;";
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2573 const char[] appendSixBits = "d = (d << 6) | *p & 0b0011_1111;";
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2574
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2575 // Decode
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2576 if ((d & 0b1110_0000) == 0b1100_0000)
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2577 { // 110xxxxx 10xxxxxx
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2578 d &= 0b0001_1111;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2579 mixin(appendSixBits);
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2580 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2581 else if ((d & 0b1111_0000) == 0b1110_0000)
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2582 { // 1110xxxx 10xxxxxx 10xxxxxx
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2583 d &= 0b0000_1111;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2584 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2585 checkNextByte ~ appendSixBits);
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2586 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2587 else if ((d & 0b1111_1000) == 0b1111_0000)
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2588 { // 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2589 d &= 0b0000_0111;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2590 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2591 checkNextByte ~ appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2592 checkNextByte ~ appendSixBits);
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2593 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2594 else
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2595 // 5 and 6 byte UTF-8 sequences are not allowed yet.
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2596 // 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2597 // 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2598 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2599
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2600 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2601
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2602 if (!isValidChar(d))
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2603 {
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2604 Lerr:
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2605 // Three cases:
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2606 // *) the UTF-8 sequence was successfully decoded but the resulting
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2607 // character is invalid.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2608 // p points to last trail byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2609 // *) the UTF-8 sequence is overlong.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2610 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2611 // *) the UTF-8 sequence has more than 4 bytes or starts with
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2612 // a trail byte.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2613 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2614 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2615 // Move to next ASCII character or lead byte of a UTF-8 sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2616 while (p < (end-1) && isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2617 ++p;
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2618 --p;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2619 assert(!isTrailByte(p[1]));
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2620 Lerr2:
430
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 426
diff changeset
2621 d = REPLACEMENT_CHAR;
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2622 error(this.p, MID.InvalidUTF8Sequence);
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2623 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2624
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2625 this.p = p;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2626 return d;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2627 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2628
519
9ebc799c7dc5 Fixes in dil.Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 514
diff changeset
2629 static void encodeUTF8(ref char[] str, dchar d)
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2630 {
430
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 426
diff changeset
2631 assert(!isascii(d), "check for ASCII char before calling encodeUTF8().");
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2632 assert(isValidChar(d), "check if character is valid before calling encodeUTF8().");
430
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 426
diff changeset
2633
519
9ebc799c7dc5 Fixes in dil.Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 514
diff changeset
2634 char[6] b = void;
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2635 if (d < 0x800)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2636 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2637 b[0] = 0xC0 | (d >> 6);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2638 b[1] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2639 str ~= b[0..2];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2640 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2641 else if (d < 0x10000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2642 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2643 b[0] = 0xE0 | (d >> 12);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2644 b[1] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2645 b[2] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2646 str ~= b[0..3];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2647 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2648 else if (d < 0x200000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2649 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2650 b[0] = 0xF0 | (d >> 18);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2651 b[1] = 0x80 | ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2652 b[2] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2653 b[3] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2654 str ~= b[0..4];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2655 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2656 /+ // There are no 5 and 6 byte UTF-8 sequences yet.
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2657 else if (d < 0x4000000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2658 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2659 b[0] = 0xF8 | (d >> 24);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2660 b[1] = 0x80 | ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2661 b[2] = 0x80 | ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2662 b[3] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2663 b[4] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2664 str ~= b[0..5];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2665 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2666 else if (d < 0x80000000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2667 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2668 b[0] = 0xFC | (d >> 30);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2669 b[1] = 0x80 | ((d >> 24) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2670 b[2] = 0x80 | ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2671 b[3] = 0x80 | ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2672 b[4] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2673 b[5] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2674 str ~= b[0..6];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2675 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2676 +/
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2677 else
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2678 assert(0);
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2679 }
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
2680 }
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2681
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2682 unittest
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2683 {
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
2684 Stdout("Testing Lexer.\n");
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2685 struct Pair
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2686 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2687 char[] tokenText;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2688 TOK kind;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2689 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2690 static Pair[] pairs = [
494
9a7ca8c56e59 Refactored a few things in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
2691 {"#!äöüß", TOK.Shebang}, {"\n", TOK.Newline},
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2692 {"//çay", TOK.Comment}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2693 {"&", TOK.AndBinary},
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2694 {"/*çağ*/", TOK.Comment}, {"&&", TOK.AndLogical},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2695 {"/+çak+/", TOK.Comment}, {"&=", TOK.AndAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2696 {">", TOK.Greater}, {"+", TOK.Plus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2697 {">=", TOK.GreaterEqual}, {"++", TOK.PlusPlus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2698 {">>", TOK.RShift}, {"+=", TOK.PlusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2699 {">>=", TOK.RShiftAssign}, {"-", TOK.Minus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2700 {">>>", TOK.URShift}, {"--", TOK.MinusMinus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2701 {">>>=", TOK.URShiftAssign}, {"-=", TOK.MinusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2702 {"<", TOK.Less}, {"=", TOK.Assign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2703 {"<=", TOK.LessEqual}, {"==", TOK.Equal},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2704 {"<>", TOK.LorG}, {"~", TOK.Tilde},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2705 {"<>=", TOK.LorEorG}, {"~=", TOK.CatAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2706 {"<<", TOK.LShift}, {"*", TOK.Mul},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2707 {"<<=", TOK.LShiftAssign}, {"*=", TOK.MulAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2708 {"!", TOK.Not}, {"/", TOK.Div},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2709 {"!=", TOK.NotEqual}, {"/=", TOK.DivAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2710 {"!<", TOK.UorGorE}, {"^", TOK.Xor},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2711 {"!>", TOK.UorLorE}, {"^=", TOK.XorAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2712 {"!<=", TOK.UorG}, {"%", TOK.Mod},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2713 {"!>=", TOK.UorL}, {"%=", TOK.ModAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2714 {"!<>", TOK.UorE}, {"(", TOK.LParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2715 {"!<>=", TOK.Unordered}, {")", TOK.RParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2716 {".", TOK.Dot}, {"[", TOK.LBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2717 {"..", TOK.Slice}, {"]", TOK.RBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2718 {"...", TOK.Ellipses}, {"{", TOK.LBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2719 {"|", TOK.OrBinary}, {"}", TOK.RBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2720 {"||", TOK.OrLogical}, {":", TOK.Colon},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2721 {"|=", TOK.OrAssign}, {";", TOK.Semicolon},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2722 {"?", TOK.Question}, {",", TOK.Comma},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2723 {"$", TOK.Dollar}, {"cam", TOK.Identifier},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2724 {"çay", TOK.Identifier}, {".0", TOK.Float64},
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2725 {"0", TOK.Int32}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2726 {"\r", TOK.Newline}, {"\r\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2727 {"\u2028", TOK.Newline}, {"\u2029", TOK.Newline}
40
9d5ceb0f8be9 - Added more tokens for testing.
aziz
parents: 39
diff changeset
2728 ];
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2729
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2730 char[] src;
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2731
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2732 // Join all token texts into a single string.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2733 foreach (i, pair; pairs)
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2734 if (pair.kind == TOK.Comment && pair.tokenText[1] == '/' || // Line comment.
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2735 pair.kind == TOK.Shebang)
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2736 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2737 assert(pairs[i+1].kind == TOK.Newline); // Must be followed by a newline.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2738 src ~= pair.tokenText;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2739 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2740 else
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2741 src ~= pair.tokenText ~ " ";
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2742
764
4579e8505d5e Fixed unittests and removed dil.File.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 763
diff changeset
2743 auto lx = new Lexer(new SourceText("", src));
350
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2744 auto token = lx.getTokens();
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2745
350
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2746 uint i;
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2747 assert(token == lx.head);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2748 assert(token.next.kind == TOK.Newline);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2749 token = token.next.next;
350
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2750 do
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2751 {
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2752 assert(i < pairs.length);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2753 assert(token.srcText == pairs[i].tokenText, Format("Scanned '{0}' but expected '{1}'", token.srcText, pairs[i].tokenText));
350
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2754 ++i;
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2755 token = token.next;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2756 } while (token.kind != TOK.EOF)
41
2b7be1d67d4d - Optimized scanner of block comments.
aziz
parents: 40
diff changeset
2757 }
55
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2758
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2759 unittest
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2760 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2761 Stdout("Testing method Lexer.peek()\n");
764
4579e8505d5e Fixed unittests and removed dil.File.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 763
diff changeset
2762 auto sourceText = new SourceText("", "unittest { }");
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2763 auto lx = new Lexer(sourceText, null);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2764
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2765 auto next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2766 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2767 assert(next.kind == TOK.Newline);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2768 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2769 assert(next.kind == TOK.Unittest);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2770 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2771 assert(next.kind == TOK.LBrace);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2772 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2773 assert(next.kind == TOK.RBrace);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2774 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2775 assert(next.kind == TOK.EOF);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2776
764
4579e8505d5e Fixed unittests and removed dil.File.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 763
diff changeset
2777 lx = new Lexer(new SourceText("", ""));
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2778 next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2779 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2780 assert(next.kind == TOK.Newline);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2781 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2782 assert(next.kind == TOK.EOF);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2783 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2784
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2785 unittest
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2786 {
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2787 // Numbers unittest
68
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2788 // 0L 0ULi 0_L 0_UL 0x0U 0x0p2 0_Fi 0_e2 0_F 0_i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2789 // 0u 0U 0uL 0UL 0L 0LU 0Lu
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2790 // 0Li 0f 0F 0fi 0Fi 0i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2791 // 0b_1_LU 0b1000u
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2792 // 0x232Lu
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2793 }