Mercurial > projects > dil

annotate trunk/src/dil/lexer/Lexer.d @ 737:f88b5285b86b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Implemented DDocEmitter. Fixed quite a few bugs.
author Aziz K?ksal <aziz.koeksal@gmail.com>
date Sat, 09 Feb 2008 02:00:20 +0100
parents b6c6baa41267
children fc8f0e61bc42
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
1 /++
8ba2570de175 Initial import.
aziz
parents:
diff changeset
2 Author: Aziz Köksal
249
32d354584b28 - Upgraded license notices to GPL3.
aziz
parents: 239
diff changeset
3 License: GPL3
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
4 +/
576
0df647660e76 Moved Lexer.d to new package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 552
diff changeset
5 module dil.lexer.Lexer;
577
9e811db780a6 Moved LexerFuncs.d to package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 576
diff changeset
6
596
39fac5531b85 Moved dil.Token to dil.lexer.Token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
7 import dil.lexer.Token;
599
c4cdea3a65ba Moved dil.Keywords to dil.lexer.Keywords.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 596
diff changeset
8 import dil.lexer.Keywords;
600
041eae272362 Moved dil.Identifier to dil.lexer.Identifier.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 599
diff changeset
9 import dil.lexer.Identifier;
601
9f96fd51cb41 Moved dil.IdTable to dil.lexer.IdTable.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
10 import dil.lexer.IdTable;
327
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
11 import dil.Information;
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
12 import dil.Messages;
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
13 import dil.HtmlEntities;
464
325714d8aa6c Added new module with compiler constants to resolve import cycle.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 434
diff changeset
14 import dil.CompilerInfo;
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
15 import dil.Unicode;
737
f88b5285b86b Implemented DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 715
diff changeset
16 import common;
f88b5285b86b Implemented DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 715
diff changeset
17
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
18 import tango.stdc.stdlib : strtof, strtod, strtold;
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
19 import tango.stdc.errno : errno, ERANGE;
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
20 import tango.stdc.time : time_t, time, ctime;
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
21 import tango.stdc.string : strlen;
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
22
577
9e811db780a6 Moved LexerFuncs.d to package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 576
diff changeset
23 public import dil.lexer.Funcs;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
24
499
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
25 /++
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
26 The Lexer analyzes the characters of a source text and
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
27 produces a doubly-linked list of tokens.
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
28 +/
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
29 class Lexer
8ba2570de175 Initial import.
aziz
parents:
diff changeset
30 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
31 Token* head; /// The head of the doubly linked token list.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
32 Token* tail; /// The tail of the linked list. Set in scan().
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
33 Token* token; /// Points to the current token in the token list.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
34 string text; /// The source text.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
35 char[] filePath; /// Path to the source text.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
36 char* p; /// Points to the current character in the source text.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
37 char* end; /// Points one character past the end of the source text.
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
38
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
39 // Members used for error messages:
532
50e64bab9c7a Renamed InformationManager to InfoManager.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 520
diff changeset
40 InfoManager infoMan;
514
6ddff941862a Added new error classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 510
diff changeset
41 LexerError[] errors;
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
42 /// Always points to the first character of the current line.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
43 char* lineBegin;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
44 // Token* newline; /// Current newline token.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
45 uint lineNum = 1; /// Current, actual source text line number.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
46 uint lineNum_hline; /// Line number set by #line.
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
47 uint inTokenString; /// > 0 if inside q{ }
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
48 NewlineData.FilePaths* filePaths;
103
511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too.
aziz
parents: 94
diff changeset
49
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
50 /++
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
51 Construct a Lexer object.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
52 Params:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
53 text = the UTF-8 source code.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
54 filePath = the path to the source code; used for error messages.
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
55 infoMan = the information manager (for collecting error messages.)
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
56 +/
532
50e64bab9c7a Renamed InformationManager to InfoManager.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 520
diff changeset
57 this(string text, string filePath, InfoManager infoMan = null)
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
58 {
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
59 this.filePath = filePath;
520
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
60 this.infoMan = infoMan;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
61
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
62 this.text = text;
397
c99f8aeb7b4a Empty source files are handled correctly now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 396
diff changeset
63 if (text.length == 0 || text[$-1] != 0)
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
64 {
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
65 this.text.length = this.text.length + 1;
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
66 this.text[$-1] = 0;
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
67 }
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
68
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
69 this.p = this.text.ptr;
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
70 this.end = this.p + this.text.length;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
71 this.lineBegin = this.p;
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
72
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
73 this.head = new Token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
74 this.head.kind = TOK.HEAD;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
75 this.head.start = this.head.end = this.p;
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
76 this.token = this.head;
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
77 // Initialize this.filePaths.
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
78 newFilePath(this.filePath);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
79 // Add a newline as the first token after the head.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
80 auto newline = new Token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
81 newline.kind = TOK.Newline;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
82 newline.setWhitespaceFlag();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
83 newline.start = newline.end = this.p;
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
84 newline.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
85 newline.newline.oriLineNum = 1;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
86 newline.newline.setLineNum = 0;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
87 // Link in.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
88 this.token.next = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
89 newline.prev = this.token;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
90 this.token = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
91 // this.newline = newline;
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
92 scanShebang();
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
93 }
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
94
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
95 ~this()
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
96 {
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
97 auto token = head.next;
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
98 while (token !is null)
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
99 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
100 assert(token.kind == TOK.EOF ? token == tail && token.next is null : 1);
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
101 delete token.prev;
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
102 token = token.next;
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
103 }
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
104 delete tail;
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
105 }
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
106
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
107 /++
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
108 The "shebang" may optionally appear once at the beginning of a file.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
109 Regexp: #![^\EndOfLine]*
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
110 +/
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
111 void scanShebang()
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
112 {
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
113 if (*p == '#' && p[1] == '!')
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
114 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
115 auto t = new Token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
116 t.kind = TOK.Shebang;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
117 t.setWhitespaceFlag();
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
118 t.start = p;
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
119 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
120 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
121 isascii(*p) || decodeUTF8();
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
122 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
123 this.token.next = t;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
124 t.prev = this.token;
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
125 }
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
126 }
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
127
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
128 void finalizeSpecialToken(ref Token t)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
129 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
130 assert(t.srcText[0..2] == "__");
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
131 switch (t.kind)
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
132 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
133 case TOK.FILE:
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
134 t.str = this.filePaths.setPath;
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
135 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
136 case TOK.LINE:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
137 t.uint_ = this.errorLineNumber(this.lineNum);
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
138 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
139 case TOK.DATE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
140 TOK.TIME,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
141 TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
142 time_t time_val;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
143 time(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
144 char* str = ctime(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
145 char[] time_str = str[0 .. strlen(str)];
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
146 switch (t.kind)
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
147 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
148 case TOK.DATE:
346
ce9e2e77743f - Fix: include one space when slicing time_str for __DATE__.
aziz
parents: 344
diff changeset
149 time_str = time_str[4..11] ~ time_str[20..24] ~ \0; break;
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
150 case TOK.TIME:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
151 time_str = time_str[11..19] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
152 case TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
153 time_str = time_str[0..24] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
154 default: assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
155 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
156 t.str = time_str;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
157 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
158 case TOK.VENDOR:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
159 t.str = VENDOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
160 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
161 case TOK.VERSION:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
162 t.uint_ = VERSION_MAJOR*1000 + VERSION_MINOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
163 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
164 default:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
165 assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
166 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
167 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
168
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
169 /// Sets a new file path.
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
170 void newFilePath(char[] newPath)
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
171 {
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
172 auto paths = new NewlineData.FilePaths;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
173 paths.oriPath = this.filePath;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
174 paths.setPath = newPath;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
175 this.filePaths = paths;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
176 }
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
177
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
178 private void setLineBegin(char* p)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
179 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
180 // Check that we can look behind one character.
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
181 assert((p-1) >= text.ptr && p < end);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
182 // Check that previous character is a newline.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
183 assert(isNewlineEnd(p - 1));
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
184 this.lineBegin = p;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
185 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
186
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
187 private void scanNext(ref Token* t)
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
188 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
189 assert(t !is null);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
190 if (t.next)
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
191 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
192 t = t.next;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
193 // if (t.kind == TOK.Newline)
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
194 // this.newline = t;
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
195 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
196 else if (t != this.tail)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
197 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
198 Token* new_t = new Token;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
199 scan(*new_t);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
200 new_t.prev = t;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
201 t.next = new_t;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
202 t = new_t;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
203 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
204 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
205
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
206 /// Advance t one token forward.
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
207 void peek(ref Token* t)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
208 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
209 scanNext(t);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
210 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
211
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
212 /// Advance to the next token in the source text.
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
213 TOK nextToken()
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
214 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
215 scanNext(this.token);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
216 return this.token.kind;
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
217 }
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
218
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
219 /// Returns true if p points to the last character of a Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
220 bool isNewlineEnd(char* p)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
221 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
222 if (*p == '\n' || *p == '\r')
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
223 return true;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
224 if (*p == LS[2] || *p == PS[2])
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
225 if ((p-2) >= text.ptr)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
226 if (p[-1] == LS[1] && p[-2] == LS[0])
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
227 return true;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
228 return false;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
229 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
230
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
231 /++
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
232 This is the old scan method.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
233 TODO: profile old and new to see which one is faster.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
234 +/
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
235 public void scan(ref Token t)
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
236 in
3
4bbce78bfb1e - Added TOK enum.
aziz
parents: 2
diff changeset
237 {
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
238 assert(text.ptr <= p && p < end);
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
239 }
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
240 out
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
241 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
242 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
243 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
244 }
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
245 body
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
246 {
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
247 // Scan whitespace.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
248 if (isspace(*p))
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
249 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
250 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
251 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
252 {}
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
253 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
254
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
255 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
256 uint c = *p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
257 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
258 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
259 // Newline.
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
260 switch (*p)
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
261 {
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
262 case '\r':
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
263 if (p[1] == '\n')
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
264 ++p;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
265 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
266 assert(isNewlineEnd(p));
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
267 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
268 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
269 setLineBegin(p);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
270 // this.newline = &t;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
271 t.kind = TOK.Newline;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
272 t.setWhitespaceFlag();
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
273 t.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
274 t.newline.oriLineNum = lineNum;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
275 t.newline.setLineNum = lineNum_hline;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
276 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
277 return;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
278 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
279 if (isUnicodeNewline(p))
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
280 {
412
fb31af0fda73 Added struct Location, and token2LocTable to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 411
diff changeset
281 ++p; ++p;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 411
diff changeset
282 goto case '\n';
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
283 }
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
284 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
285 // Identifier or string literal.
13
e5211758b63c - Added isidbeg() function.
aziz
parents: 12
diff changeset
286 if (isidbeg(c))
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
287 {
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
288 if (c == 'r' && p[1] == '"' && ++p)
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
289 return scanRawStringLiteral(t);
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
290 if (c == 'x' && p[1] == '"')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
291 return scanHexStringLiteral(t);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
292 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
293 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
294 if (c == 'q' && p[1] == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
295 return scanDelimitedStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
296 if (c == 'q' && p[1] == '{')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
297 return scanTokenStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
298 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
299 // Scan identifier.
12
0989206cf73c - Added code to decode Unicode characters in identifiers.
aziz
parents: 11
diff changeset
300 Lidentifier:
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
301 do
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
302 { c = *++p; }
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
303 while (isident(c) || !isascii(c) && isUnicodeAlpha())
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
304
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
305 t.end = p;
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
306
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
307 auto id = IdTable.lookup(t.srcText);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
308 t.kind = id.type;
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
309 t.ident = id;
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
310
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
311 if (t.kind == TOK.Identifier || t.isKeyword)
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
312 return;
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
313 else if (t.isSpecialToken)
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
314 finalizeSpecialToken(t);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
315 else if (t.kind == TOK.EOF)
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
316 {
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
317 tail = &t;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
318 assert(t.srcText == "__EOF__");
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
319 }
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
320 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
321 assert(0, "unexpected token type: " ~ Token.toString(t.kind));
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
322 return;
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
323 }
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
324
15
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
325 if (isdigit(c))
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
326 return scanNumber(t);
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
327
8
d4ba94a5a282 - Parsing /* */ comments now.
aziz
parents: 7
diff changeset
328 if (c == '/')
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
329 {
8
d4ba94a5a282 - Parsing /* */ comments now.
aziz
parents: 7
diff changeset
330 c = *++p;
14
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
331 switch(c)
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
332 {
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
333 case '=':
14
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
334 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
335 t.kind = TOK.DivAssign;
14
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
336 t.end = p;
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
337 return;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
338 case '+':
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
339 return scanNestedComment(t);
32
d7011daa4740 - Added missing commas after the items in the messages table.
aziz
parents: 31
diff changeset
340 case '*':
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
341 return scanBlockComment(t);
32
d7011daa4740 - Added missing commas after the items in the messages table.
aziz
parents: 31
diff changeset
342 case '/':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
343 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
344 isascii(*p) || decodeUTF8();
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
345 t.kind = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
346 t.setWhitespaceFlag();
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
347 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
348 return;
32
d7011daa4740 - Added missing commas after the items in the messages table.
aziz
parents: 31
diff changeset
349 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
350 t.kind = TOK.Div;
10
3ee65d6e39c9 - Parsing // comments now.
aziz
parents: 9
diff changeset
351 t.end = p;
3ee65d6e39c9 - Parsing // comments now.
aziz
parents: 9
diff changeset
352 return;
3ee65d6e39c9 - Parsing // comments now.
aziz
parents: 9
diff changeset
353 }
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
354 }
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
355
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
356 switch (c)
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
357 {
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
358 case '\'':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
359 return scanCharacterLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
360 case '`':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
361 return scanRawStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
362 case '"':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
363 return scanNormalStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
364 case '\\':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
365 char[] buffer;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
366 do
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
367 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
368 c = scanEscapeSequence();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
369 if (isascii(c))
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
370 buffer ~= c;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
371 else
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
372 encodeUTF8(buffer, c);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
373 } while (*p == '\\')
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
374 buffer ~= 0;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
375 t.kind = TOK.String;
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
376 t.str = buffer;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
377 t.end = p;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
378 return;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
379 case '>': /* > >= >> >>= >>> >>>= */
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
380 c = *++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
381 switch (c)
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
382 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
383 case '=':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
384 t.kind = TOK.GreaterEqual;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
385 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
386 case '>':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
387 if (p[1] == '>')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
388 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
389 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
390 if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
391 { ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
392 t.kind = TOK.URShiftAssign;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
393 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
394 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
395 t.kind = TOK.URShift;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
396 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
397 else if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
398 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
399 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
400 t.kind = TOK.RShiftAssign;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
401 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
402 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
403 t.kind = TOK.RShift;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
404 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
405 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
406 t.kind = TOK.Greater;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
407 goto Lcommon2;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
408 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
409 assert(0);
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
410 case '<': /* < <= <> <>= << <<= */
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
411 c = *++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
412 switch (c)
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
413 {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
414 case '=':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
415 t.kind = TOK.LessEqual;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
416 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
417 case '<':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
418 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
419 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
420 t.kind = TOK.LShiftAssign;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
421 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
422 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
423 t.kind = TOK.LShift;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
424 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
425 case '>':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
426 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
427 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
428 t.kind = TOK.LorEorG;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
429 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
430 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
431 t.kind = TOK.LorG;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
432 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
433 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
434 t.kind = TOK.Less;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
435 goto Lcommon2;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
436 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
437 assert(0);
37
7f3bcb97d017 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 36
diff changeset
438 case '!': /* ! !< !> !<= !>= !<> !<>= */
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
439 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
440 switch (c)
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
441 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
442 case '<':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
443 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
444 if (c == '>')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
445 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
446 if (p[1] == '=') {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
447 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
448 t.kind = TOK.Unordered;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
449 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
450 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
451 t.kind = TOK.UorE;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
452 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
453 else if (c == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
454 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
455 t.kind = TOK.UorG;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
456 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
457 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
458 t.kind = TOK.UorGorE;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
459 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
460 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
461 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
462 case '>':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
463 if (p[1] == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
464 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
465 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
466 t.kind = TOK.UorL;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
467 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
468 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
469 t.kind = TOK.UorLorE;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
470 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
471 case '=':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
472 t.kind = TOK.NotEqual;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
473 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
474 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
475 t.kind = TOK.Not;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
476 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
477 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
478 assert(0);
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
479 case '.': /* . .[0-9] .. ... */
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
480 if (p[1] == '.')
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
481 {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
482 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
483 if (p[1] == '.') {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
484 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
485 t.kind = TOK.Ellipses;
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
486 }
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
487 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
488 t.kind = TOK.Slice;
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
489 }
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
490 else if (isdigit(p[1]))
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
491 {
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
492 return scanReal(t);
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
493 }
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
494 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
495 t.kind = TOK.Dot;
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
496 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
497 case '|': /* | || |= */
23
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
498 c = *++p;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
499 if (c == '=')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
500 t.kind = TOK.OrAssign;
23
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
501 else if (c == '|')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
502 t.kind = TOK.OrLogical;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
503 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
504 t.kind = TOK.OrBinary;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
505 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
506 }
23
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
507 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
508 case '&': /* & && &= */
24
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
509 c = *++p;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
510 if (c == '=')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
511 t.kind = TOK.AndAssign;
24
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
512 else if (c == '&')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
513 t.kind = TOK.AndLogical;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
514 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
515 t.kind = TOK.AndBinary;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
516 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
517 }
24
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
518 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
519 case '+': /* + ++ += */
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
520 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
521 if (c == '=')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
522 t.kind = TOK.PlusAssign;
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
523 else if (c == '+')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
524 t.kind = TOK.PlusPlus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
525 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
526 t.kind = TOK.Plus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
527 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
528 }
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
529 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
530 case '-': /* - -- -= */
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
531 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
532 if (c == '=')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
533 t.kind = TOK.MinusAssign;
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
534 else if (c == '-')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
535 t.kind = TOK.MinusMinus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
536 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
537 t.kind = TOK.Minus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
538 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
539 }
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
540 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
541 case '=': /* = == */
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
542 if (p[1] == '=') {
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
543 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
544 t.kind = TOK.Equal;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
545 }
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
546 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
547 t.kind = TOK.Assign;
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
548 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
549 case '~': /* ~ ~= */
27
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
550 if (p[1] == '=') {
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
551 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
552 t.kind = TOK.CatAssign;
27
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
553 }
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
554 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
555 t.kind = TOK.Tilde;
27
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
556 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
557 case '*': /* * *= */
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
558 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
559 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
560 t.kind = TOK.MulAssign;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
561 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
562 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
563 t.kind = TOK.Mul;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
564 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
565 case '^': /* ^ ^= */
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
566 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
567 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
568 t.kind = TOK.XorAssign;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
569 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
570 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
571 t.kind = TOK.Xor;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
572 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
573 case '%': /* % %= */
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
574 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
575 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
576 t.kind = TOK.ModAssign;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
577 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
578 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
579 t.kind = TOK.Mod;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
580 goto Lcommon;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
581 // Single character tokens:
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
582 case '(':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
583 t.kind = TOK.LParen;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
584 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
585 case ')':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
586 t.kind = TOK.RParen;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
587 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
588 case '[':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
589 t.kind = TOK.LBracket;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
590 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
591 case ']':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
592 t.kind = TOK.RBracket;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
593 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
594 case '{':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
595 t.kind = TOK.LBrace;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
596 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
597 case '}':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
598 t.kind = TOK.RBrace;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
599 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
600 case ':':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
601 t.kind = TOK.Colon;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
602 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
603 case ';':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
604 t.kind = TOK.Semicolon;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
605 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
606 case '?':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
607 t.kind = TOK.Question;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
608 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
609 case ',':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
610 t.kind = TOK.Comma;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
611 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
612 case '$':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
613 t.kind = TOK.Dollar;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
614 Lcommon:
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
615 ++p;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
616 Lcommon2:
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
617 t.end = p;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
618 return;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
619 case '#':
360
b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence().
aziz
parents: 350
diff changeset
620 return scanSpecialTokenSequence(t);
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
621 default:
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
622 }
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
623
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
624 // Check for EOF
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
625 if (isEOF(c))
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
626 {
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
627 assert(isEOF(*p), ""~*p);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
628 t.kind = TOK.EOF;
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
629 t.end = p;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
630 tail = &t;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
631 assert(t.start == t.end);
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
632 return;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
633 }
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
634
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
635 if (!isascii(c))
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
636 {
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
637 c = decodeUTF8();
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
638 if (isUniAlpha(c))
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
639 goto Lidentifier;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
640 }
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
641
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
642 error(t.start, MID.IllegalCharacter, cast(dchar)c);
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
643
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
644 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
645 t.kind = TOK.Illegal;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
646 t.setWhitespaceFlag();
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
647 t.dchar_ = c;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
648 t.end = p;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
649 return;
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
650 }
3
4bbce78bfb1e - Added TOK enum.
aziz
parents: 2
diff changeset
651 }
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
652
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
653 template toUint(char[] T)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
654 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
655 static assert(0 < T.length && T.length <= 4);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
656 static if (T.length == 1)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
657 const uint toUint = T[0];
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
658 else
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
659 const uint toUint = (T[0] << ((T.length-1)*8)) | toUint!(T[1..$]);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
660 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
661 static assert(toUint!("\xAA\xBB\xCC\xDD") == 0xAABBCCDD);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
662
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
663 // Can't use this yet due to a bug in DMD (bug id=1534).
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
664 template case_(char[] str, TOK tok, char[] label)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
665 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
666 const char[] case_ =
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
667 `case `~toUint!(str).stringof~`:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
668
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
669 goto `~label~`;`;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
670 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
671
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
672 template case_L4(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
673 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
674 const char[] case_L4 = case_!(str, tok, "Lcommon_4");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
675 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
676
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
677 template case_L3(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
678 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
679 const char[] case_L3 = case_!(str, tok, "Lcommon_3");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
680 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
681
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
682 template case_L2(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
683 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
684 const char[] case_L2 = case_!(str, tok, "Lcommon_2");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
685 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
686
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
687 template case_L1(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
688 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
689 const char[] case_L3 = case_!(str, tok, "Lcommon");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
690 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
691
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
692 public void scan_(ref Token t)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
693 in
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
694 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
695 assert(text.ptr <= p && p < end);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
696 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
697 out
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
698 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
699 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
700 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
701 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
702 body
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
703 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
704 // Scan whitespace.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
705 if (isspace(*p))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
706 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
707 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
708 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
709 {}
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
710 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
711
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
712 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
713 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
714 // Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
715 switch (*p)
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
716 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
717 case '\r':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
718 if (p[1] == '\n')
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
719 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
720 case '\n':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
721 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
722 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
723 ++lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
724 setLineBegin(p);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
725 // this.newline = &t;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
726 t.kind = TOK.Newline;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
727 t.setWhitespaceFlag();
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
728 t.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
729 t.newline.oriLineNum = lineNum;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
730 t.newline.setLineNum = lineNum_hline;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
731 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
732 return;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
733 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
734 if (isUnicodeNewline(p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
735 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
736 ++p; ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
737 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
738 }
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
739 }
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
740
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
741 uint c = *p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
742 assert(end - p != 0);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
743 switch (end - p)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
744 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
745 case 1:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
746 goto L1character;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
747 case 2:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
748 c <<= 8; c |= p[1];
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
749 goto L2characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
750 case 3:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
751 c <<= 8; c |= p[1]; c <<= 8; c |= p[2];
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
752 goto L3characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
753 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
754 version(BigEndian)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
755 c = *cast(uint*)p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
756 else
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
757 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
758 c <<= 8; c |= p[1]; c <<= 8; c |= p[2]; c <<= 8; c |= p[3];
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
759 /+
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
760 c = *cast(uint*)p;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
761 asm
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
762 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
763 mov EDX, c;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
764 bswap EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
765 mov c, EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
766 }
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
767 +/
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
768 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
769 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
770
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
771 // 4 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
772 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
773 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
774 case toUint!(">>>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
775 t.kind = TOK.RShiftAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
776 goto Lcommon_4;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
777 case toUint!("!<>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
778 t.kind = TOK.Unordered;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
779 Lcommon_4:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
780 p += 4;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
781 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
782 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
783 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
784 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
785
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
786 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
787 L3characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
788 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
789 // 3 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
790 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
791 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
792 case toUint!(">>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
793 t.kind = TOK.RShiftAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
794 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
795 case toUint!(">>>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
796 t.kind = TOK.URShift;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
797 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
798 case toUint!("<>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
799 t.kind = TOK.LorEorG;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
800 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
801 case toUint!("<<="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
802 t.kind = TOK.LShiftAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
803 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
804 case toUint!("!<="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
805 t.kind = TOK.UorG;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
806 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
807 case toUint!("!>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
808 t.kind = TOK.UorL;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
809 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
810 case toUint!("!<>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
811 t.kind = TOK.UorE;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
812 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
813 case toUint!("..."):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
814 t.kind = TOK.Ellipses;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
815 Lcommon_3:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
816 p += 3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
817 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
818 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
819 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
820 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
821
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
822 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
823 L2characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
824 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
825 // 2 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
826 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
827 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
828 case toUint!("/+"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
829 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
830 return scanNestedComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
831 case toUint!("/*"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
832 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
833 return scanBlockComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
834 case toUint!("//"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
835 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
836 assert(*p == '/');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
837 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
838 isascii(*p) || decodeUTF8();
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
839 t.kind = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
840 t.setWhitespaceFlag();
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
841 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
842 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
843 case toUint!(">="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
844 t.kind = TOK.GreaterEqual;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
845 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
846 case toUint!(">>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
847 t.kind = TOK.RShift;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
848 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
849 case toUint!("<<"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
850 t.kind = TOK.LShift;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
851 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
852 case toUint!("<="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
853 t.kind = TOK.LessEqual;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
854 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
855 case toUint!("<>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
856 t.kind = TOK.LorG;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
857 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
858 case toUint!("!<"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
859 t.kind = TOK.UorGorE;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
860 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
861 case toUint!("!>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
862 t.kind = TOK.UorLorE;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
863 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
864 case toUint!("!="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
865 t.kind = TOK.NotEqual;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
866 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
867 case toUint!(".."):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
868 t.kind = TOK.Slice;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
869 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
870 case toUint!("&&"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
871 t.kind = TOK.AndLogical;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
872 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
873 case toUint!("&="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
874 t.kind = TOK.AndAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
875 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
876 case toUint!("||"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
877 t.kind = TOK.OrLogical;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
878 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
879 case toUint!("|="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
880 t.kind = TOK.OrAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
881 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
882 case toUint!("++"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
883 t.kind = TOK.PlusPlus;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
884 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
885 case toUint!("+="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
886 t.kind = TOK.PlusAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
887 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
888 case toUint!("--"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
889 t.kind = TOK.MinusMinus;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
890 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
891 case toUint!("-="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
892 t.kind = TOK.MinusAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
893 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
894 case toUint!("=="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
895 t.kind = TOK.Equal;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
896 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
897 case toUint!("~="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
898 t.kind = TOK.CatAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
899 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
900 case toUint!("*="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
901 t.kind = TOK.MulAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
902 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
903 case toUint!("/="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
904 t.kind = TOK.DivAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
905 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
906 case toUint!("^="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
907 t.kind = TOK.XorAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
908 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
909 case toUint!("%="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
910 t.kind = TOK.ModAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
911 Lcommon_2:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
912 p += 2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
913 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
914 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
915 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
916 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
917
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
918 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
919 L1character:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
920 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
921 assert(*p == c, Format("p={0},c={1}", *p, cast(dchar)c));
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
922 // 1 character tokens.
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
923 // TODO: consider storing the token type in ptable.
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
924 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
925 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
926 case '\'':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
927 return scanCharacterLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
928 case '`':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
929 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
930 case '"':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
931 return scanNormalStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
932 case '\\':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
933 char[] buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
934 do
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
935 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
936 c = scanEscapeSequence();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
937 if (isascii(c))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
938 buffer ~= c;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
939 else
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
940 encodeUTF8(buffer, c);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
941 } while (*p == '\\')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
942 buffer ~= 0;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
943 t.kind = TOK.String;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
944 t.str = buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
945 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
946 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
947 case '<':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
948 t.kind = TOK.Greater;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
949 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
950 case '>':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
951 t.kind = TOK.Less;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
952 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
953 case '^':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
954 t.kind = TOK.Xor;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
955 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
956 case '!':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
957 t.kind = TOK.Not;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
958 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
959 case '.':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
960 if (isdigit(p[1]))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
961 return scanReal(t);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
962 t.kind = TOK.Dot;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
963 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
964 case '&':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
965 t.kind = TOK.AndBinary;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
966 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
967 case '|':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
968 t.kind = TOK.OrBinary;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
969 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
970 case '+':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
971 t.kind = TOK.Plus;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
972 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
973 case '-':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
974 t.kind = TOK.Minus;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
975 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
976 case '=':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
977 t.kind = TOK.Assign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
978 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
979 case '~':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
980 t.kind = TOK.Tilde;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
981 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
982 case '*':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
983 t.kind = TOK.Mul;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
984 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
985 case '/':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
986 t.kind = TOK.Div;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
987 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
988 case '%':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
989 t.kind = TOK.Mod;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
990 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
991 case '(':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
992 t.kind = TOK.LParen;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
993 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
994 case ')':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
995 t.kind = TOK.RParen;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
996 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
997 case '[':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
998 t.kind = TOK.LBracket;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
999 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1000 case ']':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1001 t.kind = TOK.RBracket;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1002 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1003 case '{':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1004 t.kind = TOK.LBrace;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1005 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1006 case '}':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1007 t.kind = TOK.RBrace;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1008 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1009 case ':':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1010 t.kind = TOK.Colon;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1011 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1012 case ';':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1013 t.kind = TOK.Semicolon;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1014 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1015 case '?':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1016 t.kind = TOK.Question;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1017 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1018 case ',':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1019 t.kind = TOK.Comma;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1020 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1021 case '$':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1022 t.kind = TOK.Dollar;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1023 Lcommon:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1024 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1025 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1026 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1027 case '#':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1028 return scanSpecialTokenSequence(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1029 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1030 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1031
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1032 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1033 assert(*p == c);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1034
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1035 // TODO: consider moving isidbeg() and isdigit() up.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1036 if (isidbeg(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1037 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1038 if (c == 'r' && p[1] == '"' && ++p)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1039 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1040 if (c == 'x' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1041 return scanHexStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1042 version(D2)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1043 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1044 if (c == 'q' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1045 return scanDelimitedStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1046 if (c == 'q' && p[1] == '{')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1047 return scanTokenStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1048 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1049 // Scan identifier.
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1050 Lidentifier:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1051 do
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1052 { c = *++p; }
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
1053 while (isident(c) || !isascii(c) && isUnicodeAlpha())
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1054
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1055 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1056
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
1057 auto id = IdTable.lookup(t.srcText);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1058 t.kind = id.type;
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
1059 t.ident = id;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1060
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1061 if (t.kind == TOK.Identifier || t.isKeyword)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1062 return;
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1063 else if (t.isSpecialToken)
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1064 finalizeSpecialToken(t);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1065 else if (t.kind == TOK.EOF)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1066 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1067 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1068 assert(t.srcText == "__EOF__");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1069 }
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1070 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1071 assert(0, "unexpected token type: " ~ Token.toString(t.kind));
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1072 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1073 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1074
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1075 if (isdigit(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1076 return scanNumber(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1077
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1078 // Check for EOF
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1079 if (isEOF(c))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1080 {
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1081 assert(isEOF(*p), *p~"");
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1082 t.kind = TOK.EOF;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1083 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1084 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1085 assert(t.start == t.end);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1086 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1087 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1088
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1089 if (!isascii(c))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1090 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1091 c = decodeUTF8();
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1092 if (isUniAlpha(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1093 goto Lidentifier;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1094 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1095
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1096 error(t.start, MID.IllegalCharacter, cast(dchar)c);
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1097
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1098 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1099 t.kind = TOK.Illegal;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
1100 t.setWhitespaceFlag();
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1101 t.dchar_ = c;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1102 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1103 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1104 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1105
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1106 void scanBlockComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1107 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1108 assert(p[-1] == '/' && *p == '*');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1109 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1110 auto tokenLineBegin = lineBegin;
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1111 Loop:
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1112 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1113 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1114 switch (*++p)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1115 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1116 case '*':
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1117 if (p[1] != '/')
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1118 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1119 p += 2;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1120 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1121 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1122 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1123 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1124 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1125 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1126 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1127 setLineBegin(p+1);
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1128 break;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1129 default:
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1130 if (!isascii(*p))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1131 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1132 if (isUnicodeNewlineChar(decodeUTF8()))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1133 goto case '\n';
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1134 }
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1135 else if (isEOF(*p))
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1136 {
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1137 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedBlockComment);
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1138 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1139 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1140 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1141 }
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1142 t.kind = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
1143 t.setWhitespaceFlag();
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1144 t.end = p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1145 return;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1146 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1147
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1148 void scanNestedComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1149 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1150 assert(p[-1] == '/' && *p == '+');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1151 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1152 auto tokenLineBegin = lineBegin;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1153 uint level = 1;
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1154 Loop:
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1155 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1156 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1157 switch (*++p)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1158 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1159 case '/':
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1160 if (p[1] == '+')
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1161 ++p, ++level;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1162 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1163 case '+':
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1164 if (p[1] != '/')
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1165 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1166 ++p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1167 if (--level != 0)
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1168 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1169 ++p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1170 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1171 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1172 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1173 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1174 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1175 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1176 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1177 setLineBegin(p+1);
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1178 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1179 default:
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1180 if (!isascii(*p))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1181 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1182 if (isUnicodeNewlineChar(decodeUTF8()))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1183 goto case '\n';
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1184 }
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1185 else if (isEOF(*p))
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1186 {
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1187 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedNestedComment);
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1188 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1189 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1190 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1191 }
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1192 t.kind = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
1193 t.setWhitespaceFlag();
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1194 t.end = p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1195 return;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1196 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1197
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1198 char scanPostfix()
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1199 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1200 assert(p[-1] == '"' || p[-1] == '`' ||
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1201 { version(D2) return p[-1] == '}';
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1202 else return 0; }()
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1203 );
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1204 switch (*p)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1205 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1206 case 'c':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1207 case 'w':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1208 case 'd':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1209 return *p++;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1210 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1211 return 0;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1212 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1213 assert(0);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1214 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1215
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1216 void scanNormalStringLiteral(ref Token t)
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1217 {
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1218 assert(*p == '"');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1219 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1220 auto tokenLineBegin = lineBegin;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1221 t.kind = TOK.String;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1222 char[] buffer;
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1223 uint c;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1224 while (1)
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1225 {
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1226 c = *++p;
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1227 switch (c)
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1228 {
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1229 case '"':
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1230 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1231 t.pf = scanPostfix();
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1232 Lreturn:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1233 t.str = buffer ~ '\0';
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1234 t.end = p;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1235 return;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1236 case '\\':
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1237 c = scanEscapeSequence();
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1238 --p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1239 if (isascii(c))
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1240 break;
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1241 encodeUTF8(buffer, c);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1242 continue;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1243 case '\r':
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1244 if (p[1] == '\n')
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1245 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1246 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1247 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1248 c = '\n'; // Convert Newline to \n.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1249 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1250 setLineBegin(p+1);
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1251 break;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1252 case 0, _Z_:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1253 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedString);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1254 goto Lreturn;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1255 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1256 if (!isascii(c))
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1257 {
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1258 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1259 if (isUnicodeNewlineChar(c))
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1260 goto case '\n';
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1261 encodeUTF8(buffer, c);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1262 continue;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1263 }
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1264 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1265 assert(isascii(c));
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1266 buffer ~= c;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1267 }
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
1268 assert(0);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1269 }
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1270
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1271 void scanCharacterLiteral(ref Token t)
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1272 {
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1273 assert(*p == '\'');
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1274 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1275 t.kind = TOK.CharLiteral;
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1276 switch (*p)
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1277 {
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1278 case '\\':
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1279 t.dchar_ = scanEscapeSequence();
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1280 break;
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1281 case '\'':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1282 error(t.start, MID.EmptyCharacterLiteral);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1283 break;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1284 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1285 if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1286 break;
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1287 uint c = *p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1288 if (!isascii(c))
49
7f0fa15dcffc - Renamed function.
aziz
parents: 48
diff changeset
1289 c = decodeUTF8();
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1290 t.dchar_ = c;
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1291 ++p;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1292 }
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1293
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1294 if (*p == '\'')
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1295 ++p;
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1296 else
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1297 error(t.start, MID.UnterminatedCharacterLiteral);
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1298 t.end = p;
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1299 }
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1300
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1301 void scanRawStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1302 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1303 assert(*p == '`' || *p == '"' && p[-1] == 'r');
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1304 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1305 auto tokenLineBegin = lineBegin;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1306 t.kind = TOK.String;
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1307 uint delim = *p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1308 char[] buffer;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1309 uint c;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1310 while (1)
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1311 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1312 c = *++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1313 switch (c)
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1314 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1315 case '\r':
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1316 if (p[1] == '\n')
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1317 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1318 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1319 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1320 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1321 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1322 setLineBegin(p+1);
52
f65a83c27638 - Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well.
aziz
parents: 51
diff changeset
1323 break;
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1324 case '`':
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1325 case '"':
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1326 if (c == delim)
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1327 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1328 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1329 t.pf = scanPostfix();
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1330 Lreturn:
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1331 t.str = buffer ~ '\0';
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1332 t.end = p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1333 return;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1334 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1335 break;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1336 case 0, _Z_:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1337 error(tokenLineNum, tokenLineBegin, t.start,
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1338 delim == 'r' ? MID.UnterminatedRawString : MID.UnterminatedBackQuoteString);
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1339 goto Lreturn;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1340 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1341 if (!isascii(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1342 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1343 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1344 if (isUnicodeNewlineChar(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1345 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1346 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1347 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1348 }
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1349 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1350 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1351 buffer ~= c;
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1352 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1353 assert(0);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1354 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1355
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1356 void scanHexStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1357 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1358 assert(p[0] == 'x' && p[1] == '"');
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1359 t.kind = TOK.String;
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1360
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1361 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1362 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1363
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1364 uint c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1365 ubyte[] buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1366 ubyte h; // hex number
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1367 uint n; // number of hex digits
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1368
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1369 ++p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1370 assert(*p == '"');
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1371 while (1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1372 {
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1373 c = *++p;
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1374 switch (c)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1375 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1376 case '"':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1377 if (n & 1)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1378 error(tokenLineNum, tokenLineBegin, t.start, MID.OddNumberOfDigitsInHexString);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1379 ++p;
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1380 t.pf = scanPostfix();
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1381 Lreturn:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1382 t.str = cast(string) (buffer ~= 0);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1383 t.end = p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1384 return;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1385 case '\r':
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1386 if (p[1] == '\n')
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1387 ++p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1388 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1389 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1390 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1391 setLineBegin(p+1);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1392 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1393 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1394 if (ishexad(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1395 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1396 if (c <= '9')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1397 c -= '0';
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1398 else if (c <= 'F')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1399 c -= 'A' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1400 else
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1401 c -= 'a' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1402
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1403 if (n & 1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1404 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1405 h <<= 4;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1406 h |= c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1407 buffer ~= h;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1408 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1409 else
112
004d98df65af - Implemented parseInterfaceDeclaration().
aziz
parents: 105
diff changeset
1410 h = cast(ubyte)c;
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1411 ++n;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1412 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1413 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1414 else if (isspace(c))
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
1415 continue; // Skip spaces.
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
1416 else if (isEOF(c))
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1417 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1418 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedHexString);
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1419 t.pf = 0;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1420 goto Lreturn;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1421 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1422 else
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1423 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1424 auto errorAt = p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1425 if (!isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1426 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1427 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1428 if (isUnicodeNewlineChar(c))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1429 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1430 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1431 error(errorAt, MID.NonHexCharInHexString, cast(dchar)c);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1432 }
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1433 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1434 }
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1435 assert(0);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1436 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1437
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1438 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1439 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1440 void scanDelimitedStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1441 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1442 assert(p[0] == 'q' && p[1] == '"');
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1443 t.kind = TOK.String;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1444
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1445 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1446 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1447
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1448 char[] buffer;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1449 dchar opening_delim = 0, // 0 if no nested delimiter or '[', '(', '<', '{'
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1450 closing_delim; // Will be ']', ')', '>', '},
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1451 // the first character of an identifier or
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1452 // any other Unicode/ASCII character.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1453 char[] str_delim; // Identifier delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1454 uint level = 1; // Counter for nestable delimiters.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1455
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1456 ++p; ++p; // Skip q"
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1457 uint c = *p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1458 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1459 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1460 case '(':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1461 opening_delim = c;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1462 closing_delim = ')'; // c + 1
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1463 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1464 case '[', '<', '{':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1465 opening_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1466 closing_delim = c + 2; // Get to closing counterpart. Feature of ASCII table.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1467 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1468 default:
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1469 dchar scanNewline()
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1470 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1471 switch (*p)
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1472 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1473 case '\r':
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1474 if (p[1] == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1475 ++p;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1476 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1477 assert(isNewlineEnd(p));
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1478 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1479 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1480 setLineBegin(p);
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1481 return '\n';
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1482 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1483 if (isUnicodeNewline(p))
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1484 {
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
1485 ++p; ++p;
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
1486 goto case '\n';
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1487 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1488 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1489 return 0;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1490 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1491 // Skip leading newlines:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1492 while (scanNewline() != 0)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1493 {}
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1494 assert(!isNewline(p));
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1495
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1496 char* begin = p;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1497 c = *p;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1498 closing_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1499 // TODO: Check for non-printable characters?
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1500 if (!isascii(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1501 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1502 closing_delim = decodeUTF8();
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1503 if (!isUniAlpha(closing_delim))
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1504 break; // Not an identifier.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1505 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1506 else if (!isidbeg(c))
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1507 break; // Not an identifier.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1508
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1509 // Parse Identifier + EndOfLine
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1510 do
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1511 { c = *++p; }
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
1512 while (isident(c) || !isascii(c) && isUnicodeAlpha())
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1513 // Store identifier
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1514 str_delim = begin[0..p-begin];
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1515 // Scan newline
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1516 if (scanNewline() == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1517 --p; // Go back one because of "c = *++p;" in main loop.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1518 else
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1519 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1520 // TODO: error(p, MID.ExpectedNewlineAfterIdentDelim);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1521 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1522 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1523
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1524 bool checkStringDelim(char* p)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1525 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1526 assert(str_delim.length != 0);
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1527 if (buffer[$-1] == '\n' && // Last character copied to buffer must be '\n'.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1528 end-p >= str_delim.length && // Check remaining length.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1529 p[0..str_delim.length] == str_delim) // Compare.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1530 return true;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1531 return false;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1532 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1533
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1534 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1535 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1536 c = *++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1537 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1538 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1539 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1540 if (p[1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1541 ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1542 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1543 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1544 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1545 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1546 setLineBegin(p+1);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1547 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1548 case 0, _Z_:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1549 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedDelimitedString);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1550 goto Lreturn3;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1551 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1552 if (!isascii(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1553 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1554 auto begin = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1555 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1556 if (isUnicodeNewlineChar(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1557 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1558 if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1559 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1560 if (str_delim.length)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1561 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1562 if (checkStringDelim(begin))
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1563 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1564 p = begin + str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1565 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1566 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1567 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1568 else
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1569 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1570 assert(level == 1);
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1571 --level;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1572 goto Lreturn;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1573 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1574 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1575 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1576 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1577 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1578 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1579 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1580 if (c == opening_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1581 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1582 else if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1583 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1584 if (str_delim.length)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1585 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1586 if (checkStringDelim(p))
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1587 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1588 p += str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1589 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1590 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1591 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1592 else if (--level == 0)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1593 goto Lreturn;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1594 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1595 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1596 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1597 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1598 buffer ~= c;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1599 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1600 Lreturn: // Character delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1601 assert(c == closing_delim);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1602 assert(level == 0);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1603 ++p; // Skip closing delimiter.
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1604 Lreturn2: // String delimiter.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1605 if (*p == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1606 ++p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1607 else
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1608 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1609 // TODO: error(p, MID.ExpectedDblQuoteAfterDelim, str_delim.length ? str_delim : closing_delim~"");
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1610 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1611
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1612 t.pf = scanPostfix();
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1613 Lreturn3: // Error.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1614 t.str = buffer ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1615 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1616 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1617
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1618 void scanTokenStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1619 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1620 assert(p[0] == 'q' && p[1] == '{');
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1621 t.kind = TOK.String;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1622
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1623 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1624 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1625
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1626 // A guard against changes to particular members:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1627 // this.lineNum_hline and this.errorPath
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1628 ++inTokenString;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1629
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1630 uint lineNum = this.lineNum;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1631 uint level = 1;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1632
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1633 ++p; ++p; // Skip q{
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1634
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1635 auto prev_t = &t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1636 Token* token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1637 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1638 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1639 token = new Token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1640 scan(*token);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1641 // Save the tokens in a doubly linked list.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1642 // Could be useful for various tools.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1643 token.prev = prev_t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1644 prev_t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1645 prev_t = token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1646 switch (token.kind)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1647 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1648 case TOK.LBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1649 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1650 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1651 case TOK.RBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1652 if (--level == 0)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1653 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1654 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1655 t.next = null;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1656 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1657 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1658 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1659 case TOK.EOF:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1660 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedTokenString);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1661 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1662 t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1663 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1664 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1665 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1666 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1667 break; // Exit loop.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1668 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1669
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1670 assert(token.kind == TOK.RBrace || token.kind == TOK.EOF);
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1671 assert(token.kind == TOK.RBrace && t.next is null ||
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1672 token.kind == TOK.EOF && t.next !is null);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1673
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1674 char[] buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1675 // token points to } or EOF
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1676 if (token.kind == TOK.EOF)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1677 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1678 t.end = token.start;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1679 buffer = t.srcText[2..$].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1680 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1681 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1682 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1683 // Assign to buffer before scanPostfix().
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1684 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1685 buffer = t.srcText[2..$-1].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1686 t.pf = scanPostfix();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1687 t.end = p; // Assign again because of postfix.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1688 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1689 // Convert newlines to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1690 if (lineNum != this.lineNum)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1691 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1692 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1693 uint i, j;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1694 for (; i < buffer.length; ++i)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1695 switch (buffer[i])
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1696 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1697 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1698 if (buffer[i+1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1699 ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1700 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1701 assert(isNewlineEnd(buffer.ptr + i));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1702 buffer[j++] = '\n'; // Convert Newline to '\n'.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1703 break;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1704 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1705 if (isUnicodeNewline(buffer.ptr + i))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1706 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1707 ++i; ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1708 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1709 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1710 buffer[j++] = buffer[i]; // Copy.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1711 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1712 buffer.length = j; // Adjust length.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1713 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1714 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1715 t.str = buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1716
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1717 --inTokenString;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1718 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1719 } // version(D2)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1720
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1721 dchar scanEscapeSequence()
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1722 out(result)
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
1723 { assert(isValidChar(result)); }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1724 body
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1725 {
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1726 assert(*p == '\\');
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1727
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1728 auto sequenceStart = p; // Used for error reporting.
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1729
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1730 ++p;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1731 uint c = char2ev(*p);
82
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1732 if (c)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1733 {
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1734 ++p;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1735 return c;
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1736 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1737
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1738 uint digits = 2;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1739
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1740 switch (*p)
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1741 {
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1742 case 'x':
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1743 assert(c == 0);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1744 while (1)
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1745 {
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1746 ++p;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1747 if (ishexad(*p))
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1748 {
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1749 c *= 16;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1750 if (*p <= '9')
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1751 c += *p - '0';
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1752 else if (*p <= 'F')
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1753 c += *p - 'A' + 10;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1754 else
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1755 c += *p - 'a' + 10;
82
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1756
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1757 if (!--digits)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1758 {
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1759 ++p;
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
1760 if (isValidChar(c))
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1761 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1762
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1763 error(sequenceStart, MID.InvalidUnicodeEscapeSequence, sequenceStart[0..p-sequenceStart]);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1764 break;
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1765 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1766 continue;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1767 }
432
3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 430
diff changeset
1768
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1769 error(sequenceStart, MID.InsufficientHexDigits);
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1770 break;
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1771 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1772 break;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1773 case 'u':
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1774 digits = 4;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1775 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1776 case 'U':
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1777 digits = 8;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1778 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1779 default:
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1780 if (isoctal(*p))
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1781 {
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1782 assert(c == 0);
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1783 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1784 ++p;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1785 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1786 return c;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1787 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1788 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1789 ++p;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1790 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1791 return c;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1792 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1793 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1794 ++p;
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1795 return c; // Return valid escape value.
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1796 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1797 else if(*p == '&')
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1798 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1799 if (isalpha(*++p))
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1800 {
272
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1801 auto begin = p;
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1802 while (isalnum(*++p))
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1803 {}
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1804
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1805 if (*p == ';')
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1806 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1807 // Pass entity excluding '&' and ';'.
272
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1808 c = entity2Unicode(begin[0..p - begin]);
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1809 ++p; // Skip ;
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1810 if (c != 0xFFFF)
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1811 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1812 else
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1813 error(sequenceStart, MID.UndefinedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
272
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1814 }
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1815 else
432
3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 430
diff changeset
1816 error(sequenceStart, MID.UnterminatedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1817 }
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1818 else
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1819 error(sequenceStart, MID.InvalidBeginHTMLEntity);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1820 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1821 else if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1822 error(sequenceStart, MID.UndefinedEscapeSequence,
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
1823 isEOF(*p) ? `\EOF` : `\NewLine`);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1824 else
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1825 {
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1826 char[] str = `\`;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1827 if (isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1828 str ~= *p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1829 else
387
ad0cbd1c8881 Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 386
diff changeset
1830 encodeUTF8(str, decodeUTF8());
ad0cbd1c8881 Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 386
diff changeset
1831 ++p;
ad0cbd1c8881 Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 386
diff changeset
1832 // TODO: check for unprintable character?
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1833 error(sequenceStart, MID.UndefinedEscapeSequence, str);
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1834 }
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1835 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1836 return REPLACEMENT_CHAR; // Error: return replacement character.
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1837 }
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1838
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1839 /*
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1840 IntegerLiteral:= (Dec|Hex|Bin|Oct)Suffix?
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1841 Dec:= (0|[1-9][0-9_]*)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1842 Hex:= 0[xX] HexDigits
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1843 Bin:= 0[bB][01_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1844 Oct:= 0[0-7_]+
68
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
1845 Suffix:= (L[uU]?|[uU]L?)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1846 HexDigits:= [0-9a-zA-Z_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1847
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1848 Invalid: "0b_", "0x_", "._"
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1849 */
15
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
1850 void scanNumber(ref Token t)
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
1851 {
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1852 ulong ulong_;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1853 bool overflow;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1854 bool isDecimal;
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1855 size_t digits;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1856
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1857 if (*p != '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1858 goto LscanInteger;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1859 ++p; // skip zero
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1860 // check for xX bB ...
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1861 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1862 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1863 case 'x','X':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1864 goto LscanHex;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1865 case 'b','B':
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1866 goto LscanBinary;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1867 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1868 if (p[1] == 'i')
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1869 goto LscanReal; // 0Li
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1870 break; // 0L
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1871 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1872 if (p[1] == '.')
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1873 break; // 0..
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1874 // 0.
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1875 case 'i','f','F', // Imaginary and float literal suffixes.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1876 'e', 'E': // Float exponent.
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1877 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1878 default:
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1879 if (*p == '_')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1880 goto LscanOctal; // 0_
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1881 else if (isdigit(*p))
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1882 {
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1883 if (*p == '8' || *p == '9')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1884 goto Loctal_hasDecimalDigits; // 08 or 09
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1885 else
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1886 goto Loctal_enter_loop; // 0[0-7]
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1887 }
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1888 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1889
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1890 // Number 0
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1891 assert(p[-1] == '0');
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1892 assert(*p != '_' && !isdigit(*p));
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1893 assert(ulong_ == 0);
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1894 isDecimal = true;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1895 goto Lfinalize;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1896
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1897 LscanInteger:
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1898 assert(*p != 0 && isdigit(*p));
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1899 isDecimal = true;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1900 goto Lenter_loop_int;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1901 while (1)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1902 {
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1903 if (*++p == '_')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1904 continue;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1905 if (!isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1906 break;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1907 Lenter_loop_int:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1908 if (ulong_ < ulong.max/10 || (ulong_ == ulong.max/10 && *p <= '5'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1909 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1910 ulong_ *= 10;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1911 ulong_ += *p - '0';
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1912 continue;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1913 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1914 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1915 overflow = true;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1916 while (isdigit(*++p)) {}
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1917 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1918 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1919
61
512cd2248dfc - Fix: issueing error on hexadecimal number overflow.
aziz
parents: 60
diff changeset
1920 // The number could be a float, so check overflow below.
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1921 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1922 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1923 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1924 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1925 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1926 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1927 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1928 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1929 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1930 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1931 goto LscanReal;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1932 default:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1933 }
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1934
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1935 if (overflow)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1936 error(t.start, MID.OverflowDecimalNumber);
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1937
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1938 assert((isdigit(p[-1]) || p[-1] == '_') && !isdigit(*p) && *p != '_');
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1939 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1940
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1941 LscanHex:
61
512cd2248dfc - Fix: issueing error on hexadecimal number overflow.
aziz
parents: 60
diff changeset
1942 assert(digits == 0);
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
1943 assert(*p == 'x' || *p == 'X');
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1944 while (1)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1945 {
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1946 if (*++p == '_')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1947 continue;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1948 if (!ishexad(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1949 break;
61
512cd2248dfc - Fix: issueing error on hexadecimal number overflow.
aziz
parents: 60
diff changeset
1950 ++digits;
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1951 ulong_ *= 16;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1952 if (*p <= '9')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1953 ulong_ += *p - '0';
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1954 else if (*p <= 'F')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1955 ulong_ += *p - 'A' + 10;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1956 else
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1957 ulong_ += *p - 'a' + 10;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1958 }
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1959
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
1960 assert(ishexad(p[-1]) || p[-1] == '_' || p[-1] == 'x' || p[-1] == 'X');
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1961 assert(!ishexad(*p) && *p != '_');
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1962
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1963 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1964 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1965 case '.':
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1966 if (p[1] == '.')
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1967 break;
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1968 case 'p', 'P':
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1969 return scanHexReal(t);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1970 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1971 }
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1972
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1973 if (digits == 0 || digits > 16)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1974 error(t.start, digits == 0 ? MID.NoDigitsInHexNumber : MID.OverflowHexNumber);
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1975
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1976 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1977
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1978 LscanBinary:
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1979 assert(digits == 0);
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
1980 assert(*p == 'b' || *p == 'B');
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1981 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1982 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1983 if (*++p == '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1984 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1985 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1986 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1987 }
367
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1988 else if (*p == '1')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1989 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1990 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1991 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1992 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1993 }
367
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1994 else if (*p == '_')
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1995 continue;
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1996 else
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1997 break;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1998 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1999
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2000 if (digits == 0 || digits > 64)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2001 error(t.start, digits == 0 ? MID.NoDigitsInBinNumber : MID.OverflowBinaryNumber);
59
3e594725899a - Issuing error when no digits were found in hex and binary numbers.
aziz
parents: 58
diff changeset
2002
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
2003 assert(p[-1] == '0' || p[-1] == '1' || p[-1] == '_' || p[-1] == 'b' || p[-1] == 'B', p[-1] ~ "");
367
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
2004 assert( !(*p == '0' || *p == '1' || *p == '_') );
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2005 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2006
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2007 LscanOctal:
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2008 assert(*p == '_');
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2009 while (1)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2010 {
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2011 if (*++p == '_')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2012 continue;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2013 if (!isoctal(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2014 break;
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2015 Loctal_enter_loop:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2016 if (ulong_ < ulong.max/2 || (ulong_ == ulong.max/2 && *p <= '1'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2017 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2018 ulong_ *= 8;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2019 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2020 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2021 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2022 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2023 overflow = true;
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2024 while (isoctal(*++p)) {}
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2025 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2026 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2027
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2028 bool hasDecimalDigits;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2029 if (isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2030 {
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2031 Loctal_hasDecimalDigits:
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2032 hasDecimalDigits = true;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2033 while (isdigit(*++p)) {}
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2034 }
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2035
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2036 // The number could be a float, so check errors below.
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2037 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2038 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2039 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2040 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2041 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2042 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2043 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2044 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2045 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2046 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2047 goto LscanReal;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2048 default:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2049 }
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2050
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2051 if (hasDecimalDigits)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2052 error(t.start, MID.OctalNumberHasDecimals);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2053
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2054 if (overflow)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2055 error(t.start, MID.OverflowOctalNumber);
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2056 // goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2057
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2058 Lfinalize:
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2059 enum Suffix
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2060 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2061 None = 0,
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2062 Unsigned = 1,
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2063 Long = 2
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2064 }
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2065
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2066 // Scan optional suffix: L, Lu, LU, u, uL, U or UL.
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2067 Suffix suffix;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2068 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2069 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2070 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2071 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2072 case 'L':
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2073 if (suffix & Suffix.Long)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2074 break;
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2075 suffix |= Suffix.Long;
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2076 ++p;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2077 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2078 case 'u', 'U':
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2079 if (suffix & Suffix.Unsigned)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2080 break;
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2081 suffix |= Suffix.Unsigned;
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2082 ++p;
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2083 continue;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2084 default:
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2085 break;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2086 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2087 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2088 }
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2089
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2090 // Determine type of Integer.
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2091 switch (suffix)
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2092 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2093 case Suffix.None:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2094 if (ulong_ & 0x8000_0000_0000_0000)
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2095 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2096 if (isDecimal)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2097 error(t.start, MID.OverflowDecimalSign);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2098 t.kind = TOK.Uint64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2099 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2100 else if (ulong_ & 0xFFFF_FFFF_0000_0000)
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2101 t.kind = TOK.Int64;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2102 else if (ulong_ & 0x8000_0000)
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2103 t.kind = isDecimal ? TOK.Int64 : TOK.Uint32;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2104 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2105 t.kind = TOK.Int32;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2106 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2107 case Suffix.Unsigned:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2108 if (ulong_ & 0xFFFF_FFFF_0000_0000)
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2109 t.kind = TOK.Uint64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2110 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2111 t.kind = TOK.Uint32;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2112 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2113 case Suffix.Long:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2114 if (ulong_ & 0x8000_0000_0000_0000)
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2115 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2116 if (isDecimal)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2117 error(t.start, MID.OverflowDecimalSign);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2118 t.kind = TOK.Uint64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2119 }
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2120 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2121 t.kind = TOK.Int64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2122 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2123 case Suffix.Unsigned | Suffix.Long:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2124 t.kind = TOK.Uint64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2125 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2126 default:
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2127 assert(0);
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2128 }
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2129 t.ulong_ = ulong_;
15
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
2130 t.end = p;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2131 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2132 LscanReal:
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2133 scanReal(t);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2134 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2135 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2136
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2137 /*
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2138 FloatLiteral:= Float[fFL]?i?
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2139 Float:= DecFloat | HexFloat
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2140 DecFloat:= ([0-9][0-9_]*[.][0-9_]*DecExponent?) | [.][0-9][0-9_]*DecExponent? | [0-9][0-9_]*DecExponent
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2141 DecExponent:= [eE][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2142 HexFloat:= 0[xX](HexDigits[.]HexDigits | [.][0-9a-zA-Z]HexDigits? | HexDigits)HexExponent
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2143 HexExponent:= [pP][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2144 */
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2145 void scanReal(ref Token t)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2146 {
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2147 if (*p == '.')
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2148 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2149 assert(p[1] != '.');
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2150 // This function was called by scan() or scanNumber().
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2151 while (isdigit(*++p) || *p == '_') {}
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2152 }
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2153 else
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2154 // This function was called by scanNumber().
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2155 assert(delegate ()
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2156 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2157 switch (*p)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2158 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2159 case 'L':
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2160 if (p[1] != 'i')
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2161 return false;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2162 case 'i', 'f', 'F', 'e', 'E':
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2163 return true;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2164 default:
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2165 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2166 return false;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2167 }()
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2168 );
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2169
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2170 // Scan exponent.
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2171 if (*p == 'e' || *p == 'E')
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2172 {
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2173 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2174 if (*p == '-' || *p == '+')
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2175 ++p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2176 if (isdigit(*p))
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2177 while (isdigit(*++p) || *p == '_') {}
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2178 else
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2179 error(t.start, MID.FloatExpMustStartWithDigit);
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2180 }
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2181
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2182 // Copy whole number and remove underscores from buffer.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2183 char[] buffer = t.start[0..p-t.start].dup;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2184 uint j;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2185 foreach (c; buffer)
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2186 if (c != '_')
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2187 buffer[j++] = c;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2188 buffer.length = j; // Adjust length.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2189 buffer ~= 0; // Terminate for C functions.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2190
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2191 finalizeFloat(t, buffer);
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2192 }
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2193
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2194 void scanHexReal(ref Token t)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2195 {
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2196 assert(*p == '.' || *p == 'p' || *p == 'P');
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2197 MID mid;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2198 if (*p == '.')
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2199 while (ishexad(*++p) || *p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2200 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2201 // Decimal exponent is required.
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2202 if (*p != 'p' && *p != 'P')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2203 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2204 mid = MID.HexFloatExponentRequired;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2205 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2206 }
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2207 // Scan exponent
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2208 assert(*p == 'p' || *p == 'P');
393
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2209 ++p;
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2210 if (*p == '+' || *p == '-')
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2211 ++p;
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2212 if (!isdigit(*p))
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2213 {
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2214 mid = MID.HexFloatExpMustStartWithDigit;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2215 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2216 }
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2217 while (isdigit(*++p) || *p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2218 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2219 // Copy whole number and remove underscores from buffer.
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2220 char[] buffer = t.start[0..p-t.start].dup;
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2221 uint j;
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2222 foreach (c; buffer)
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2223 if (c != '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2224 buffer[j++] = c;
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2225 buffer.length = j; // Adjust length.
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2226 buffer ~= 0; // Terminate for C functions.
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2227 finalizeFloat(t, buffer);
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2228 return;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2229 Lerr:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2230 t.kind = TOK.Float32;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2231 t.end = p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2232 error(t.start, mid);
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2233 }
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2234
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2235 void finalizeFloat(ref Token t, string buffer)
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2236 {
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2237 assert(buffer[$-1] == 0);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2238 // Float number is well-formed. Check suffixes and do conversion.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2239 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2240 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2241 case 'f', 'F':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2242 t.kind = TOK.Float32;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2243 t.float_ = strtof(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2244 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2245 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2246 case 'L':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2247 t.kind = TOK.Float80;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2248 t.real_ = strtold(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2249 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2250 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2251 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2252 t.kind = TOK.Float64;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2253 t.double_ = strtod(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2254 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2255 if (*p == 'i')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2256 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2257 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2258 t.kind += 3; // Switch to imaginary counterpart.
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2259 assert(t.kind == TOK.Imaginary32 ||
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2260 t.kind == TOK.Imaginary64 ||
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2261 t.kind == TOK.Imaginary80);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2262 }
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
2263 if (errno() == ERANGE)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2264 error(t.start, MID.OverflowFloatNumber);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2265 t.end = p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2266 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2267
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2268 /// Scan special token: #line Integer [Filespec] EndOfLine
360
b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence().
aziz
parents: 350
diff changeset
2269 void scanSpecialTokenSequence(ref Token t)
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2270 {
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
2271 assert(*p == '#');
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2272 t.kind = TOK.HashLine;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
2273 t.setWhitespaceFlag();
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2274
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2275 MID mid;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2276 auto errorAtColumn = p;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2277
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2278 ++p;
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2279 if (p[0] != 'l' || p[1] != 'i' || p[2] != 'n' || p[3] != 'e')
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2280 {
370
ae4afb66768f - Renamed findModule() to findModulePath().
aziz
parents: 367
diff changeset
2281 mid = MID.ExpectedIdentifierSTLine;
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2282 goto Lerr;
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2283 }
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2284 p += 3;
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2285
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2286 // TODO: #line58"path/file" is legal. Require spaces?
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2287 // State.Space could be used for that purpose.
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2288 enum State
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2289 { /+Space,+/ Integer, Filespec, End }
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2290
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2291 State state = State.Integer;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2292
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2293 while (!isEndOfLine(++p))
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2294 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2295 if (isspace(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2296 continue;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2297 if (state == State.Integer)
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2298 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2299 if (!isdigit(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2300 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2301 errorAtColumn = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2302 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2303 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2304 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2305 t.tokLineNum = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2306 scan(*t.tokLineNum);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2307 if (t.tokLineNum.kind != TOK.Int32 && t.tokLineNum.kind != TOK.Uint32)
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2308 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2309 errorAtColumn = t.tokLineNum.start;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2310 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2311 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2312 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2313 --p; // Go one back because scan() advanced p past the integer.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2314 state = State.Filespec;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2315 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2316 else if (state == State.Filespec)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2317 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2318 if (*p != '"')
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2319 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2320 errorAtColumn = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2321 mid = MID.ExpectedFilespec;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2322 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2323 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2324 t.tokLineFilespec = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2325 t.tokLineFilespec.start = p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2326 t.tokLineFilespec.kind = TOK.Filespec;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
2327 t.tokLineFilespec.setWhitespaceFlag();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2328 while (*++p != '"')
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2329 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2330 if (isEndOfLine(p))
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2331 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2332 errorAtColumn = t.tokLineFilespec.start;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2333 mid = MID.UnterminatedFilespec;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2334 t.tokLineFilespec.end = p;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2335 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2336 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2337 isascii(*p) || decodeUTF8();
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2338 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2339 auto start = t.tokLineFilespec.start +1; // +1 skips '"'
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2340 t.tokLineFilespec.str = start[0 .. p - start];
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2341 t.tokLineFilespec.end = p + 1;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2342 state = State.End;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2343 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2344 else/+ if (state == State.End)+/
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2345 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2346 mid = MID.UnterminatedSpecialToken;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2347 goto Lerr;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2348 }
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2349 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2350 assert(isEndOfLine(p));
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2351
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2352 if (state == State.Integer)
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2353 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2354 errorAtColumn = p;
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2355 mid = MID.ExpectedIntegerAfterSTLine;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2356 goto Lerr;
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2357 }
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2358
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2359 // Evaluate #line only when not in token string.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2360 if (!inTokenString && t.tokLineNum)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2361 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2362 this.lineNum_hline = this.lineNum - t.tokLineNum.uint_ + 1;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2363 if (t.tokLineFilespec)
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
2364 newFilePath(t.tokLineFilespec.str);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2365 }
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2366 t.end = p;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2367
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2368 return;
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2369 Lerr:
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2370 t.end = p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2371 error(errorAtColumn, mid);
371
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer.
aziz
parents: 370
diff changeset
2372 }
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer.
aziz
parents: 370
diff changeset
2373
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2374 /++
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2375 Insert an empty dummy token before t.
737
f88b5285b86b Implemented DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 715
diff changeset
2376
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2377 Useful in the parsing phase for representing a node in the AST
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2378 that doesn't consume an actual token from the source text.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2379 +/
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2380 Token* insertEmptyTokenBefore(Token* t)
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2381 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2382 assert(t !is null && t.prev !is null);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2383 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2384 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2385
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2386 auto prev_t = t.prev;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2387 auto new_t = new Token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2388 new_t.kind = TOK.Empty;
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2389 new_t.start = new_t.end = prev_t.end;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2390 // Link in new token.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2391 prev_t.next = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2392 new_t.prev = prev_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2393 new_t.next = t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2394 t.prev = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2395 return new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2396 }
105
df34ec47fb81 - Added getState() method and State struct to Lexer.
aziz
parents: 103
diff changeset
2397
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2398 uint errorLineNumber(uint lineNum)
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
2399 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2400 return lineNum - this.lineNum_hline;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
2401 }
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
2402
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2403 void error(char* columnPos, MID mid, ...)
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2404 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2405 error_(this.lineNum, this.lineBegin, columnPos, mid, _arguments, _argptr);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2406 }
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2407
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2408 void error(uint lineNum, char* lineBegin, char* columnPos, MID mid, ...)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2409 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2410 error_(lineNum, lineBegin, columnPos, mid, _arguments, _argptr);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2411 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2412
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2413 void error_(uint lineNum, char* lineBegin, char* columnPos, MID mid,
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2414 TypeInfo[] _arguments, void* _argptr)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2415 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2416 lineNum = this.errorLineNumber(lineNum);
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
2417 auto errorPath = this.filePaths.setPath;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2418 auto location = new Location(errorPath, lineNum, lineBegin, columnPos);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2419 auto msg = Format(_arguments, _argptr, GetMsg(mid));
520
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2420 auto error = new LexerError(location, msg);
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2421 errors ~= error;
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2422 if (infoMan !is null)
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2423 infoMan ~= error;
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2424 }
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2425
306
9c866aadcb5b - Moved code out of main() to separate functions.
aziz
parents: 272
diff changeset
2426 Token* getTokens()
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
2427 {
306
9c866aadcb5b - Moved code out of main() to separate functions.
aziz
parents: 272
diff changeset
2428 while (nextToken() != TOK.EOF)
9c866aadcb5b - Moved code out of main() to separate functions.
aziz
parents: 272
diff changeset
2429 {}
9c866aadcb5b - Moved code out of main() to separate functions.
aziz
parents: 272
diff changeset
2430 return head;
3
4bbce78bfb1e - Added TOK enum.
aziz
parents: 2
diff changeset
2431 }
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2432
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2433 /// Scan the whole text until EOF is encountered.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2434 void scanAll()
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2435 {
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2436 while (nextToken() != TOK.EOF)
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2437 {}
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2438 }
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2439
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2440 /// HEAD -> Newline -> First Token
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2441 Token* firstToken()
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2442 {
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2443 return this.head.next.next;
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2444 }
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2445
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2446 static void loadKeywords(ref Identifier[string] table)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2447 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2448 foreach(k; keywords)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2449 table[k.str] = k;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2450 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2451
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2452 /// Returns true if str is a valid D identifier.
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2453 static bool isIdentifierString(char[] str)
365
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2454 {
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2455 if (str.length == 0 || isdigit(str[0]))
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2456 return false;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2457 size_t idx;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2458 do
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2459 {
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2460 auto c = dil.Unicode.decode(str, idx);
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2461 if (c == ERROR_CHAR || !(isident(c) || !isascii(c) && isUniAlpha(c)))
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2462 return false;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2463 } while (idx < str.length)
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2464 return true;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2465 }
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2466
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2467 /// Returns true if str is a keyword or a special token (__FILE__, __LINE__ etc.)
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2468 static bool isReservedIdentifier(char[] str)
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2469 {
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2470 if (str.length == 0)
382
66477017cb95 Applied fixes to isNonReservedIdentifier().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 371
diff changeset
2471 return false;
66477017cb95 Applied fixes to isNonReservedIdentifier().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 371
diff changeset
2472
365
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2473 static Identifier[string] reserved_ids_table;
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2474 if (reserved_ids_table is null)
499
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
2475 Lexer.loadKeywords(reserved_ids_table);
365
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2476
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2477 if (!isIdentifierString(str))
365
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2478 return false;
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2479
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2480 return (str in reserved_ids_table) !is null;
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2481 }
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2482
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2483 /++
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2484 Returns true if the current character to be decoded is
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2485 a Unicode alpha character.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2486 The current pointer 'p' is not advanced if false is returned.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2487 +/
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2488 bool isUnicodeAlpha()
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2489 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2490 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2491 char* p = this.p;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2492 dchar d = *p;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2493 ++p; // Move to second byte.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2494 // Error if second byte is not a trail byte.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2495 if (!isTrailByte(*p))
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2496 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2497 // Check for overlong sequences.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2498 switch (d)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2499 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2500 case 0xE0, 0xF0, 0xF8, 0xFC:
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2501 if ((*p & d) == 0x80)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2502 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2503 default:
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2504 if ((d & 0xFE) == 0xC0) // 1100000x
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2505 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2506 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2507 const char[] checkNextByte = "if (!isTrailByte(*++p))"
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2508 " return false;";
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2509 const char[] appendSixBits = "d = (d << 6) | *p & 0b0011_1111;";
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2510 // Decode
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2511 if ((d & 0b1110_0000) == 0b1100_0000)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2512 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2513 d &= 0b0001_1111;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2514 mixin(appendSixBits);
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2515 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2516 else if ((d & 0b1111_0000) == 0b1110_0000)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2517 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2518 d &= 0b0000_1111;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2519 mixin(appendSixBits ~
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2520 checkNextByte ~ appendSixBits);
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2521 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2522 else if ((d & 0b1111_1000) == 0b1111_0000)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2523 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2524 d &= 0b0000_0111;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2525 mixin(appendSixBits ~
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2526 checkNextByte ~ appendSixBits ~
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2527 checkNextByte ~ appendSixBits);
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2528 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2529 else
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2530 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2531
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2532 assert(isTrailByte(*p));
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2533 if (!isValidChar(d) || !isUniAlpha(d))
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2534 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2535 // Only advance pointer if this is a Unicode alpha character.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2536 this.p = p;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2537 return true;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2538 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2539
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2540 /// Decodes the next UTF-8 sequence.
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2541 dchar decodeUTF8()
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2542 {
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2543 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2544 char* p = this.p;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2545 dchar d = *p;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2546
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2547 ++p; // Move to second byte.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2548 // Error if second byte is not a trail byte.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2549 if (!isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2550 goto Lerr2;
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2551
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2552 // Check for overlong sequences.
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2553 switch (d)
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2554 {
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2555 case 0xE0, // 11100000 100xxxxx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2556 0xF0, // 11110000 1000xxxx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2557 0xF8, // 11111000 10000xxx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2558 0xFC: // 11111100 100000xx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2559 if ((*p & d) == 0x80)
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2560 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2561 default:
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2562 if ((d & 0xFE) == 0xC0) // 1100000x
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2563 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2564 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2565
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2566 const char[] checkNextByte = "if (!isTrailByte(*++p))"
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2567 " goto Lerr2;";
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2568 const char[] appendSixBits = "d = (d << 6) | *p & 0b0011_1111;";
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2569
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2570 // Decode
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2571 if ((d & 0b1110_0000) == 0b1100_0000)
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2572 { // 110xxxxx 10xxxxxx
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2573 d &= 0b0001_1111;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2574 mixin(appendSixBits);
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2575 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2576 else if ((d & 0b1111_0000) == 0b1110_0000)
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2577 { // 1110xxxx 10xxxxxx 10xxxxxx
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2578 d &= 0b0000_1111;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2579 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2580 checkNextByte ~ appendSixBits);
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2581 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2582 else if ((d & 0b1111_1000) == 0b1111_0000)
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2583 { // 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2584 d &= 0b0000_0111;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2585 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2586 checkNextByte ~ appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2587 checkNextByte ~ appendSixBits);
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2588 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2589 else
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2590 // 5 and 6 byte UTF-8 sequences are not allowed yet.
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2591 // 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2592 // 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2593 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2594
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2595 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2596
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2597 if (!isValidChar(d))
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2598 {
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2599 Lerr:
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2600 // Three cases:
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2601 // *) the UTF-8 sequence was successfully decoded but the resulting
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2602 // character is invalid.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2603 // p points to last trail byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2604 // *) the UTF-8 sequence is overlong.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2605 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2606 // *) the UTF-8 sequence has more than 4 bytes or starts with
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2607 // a trail byte.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2608 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2609 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2610 // Move to next ASCII character or lead byte of a UTF-8 sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2611 while (p < (end-1) && isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2612 ++p;
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2613 --p;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2614 assert(!isTrailByte(p[1]));
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2615 Lerr2:
430
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 426
diff changeset
2616 d = REPLACEMENT_CHAR;
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2617 error(this.p, MID.InvalidUTF8Sequence);
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2618 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2619
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2620 this.p = p;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2621 return d;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2622 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2623
519
9ebc799c7dc5 Fixes in dil.Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 514
diff changeset
2624 static void encodeUTF8(ref char[] str, dchar d)
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2625 {
430
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 426
diff changeset
2626 assert(!isascii(d), "check for ASCII char before calling encodeUTF8().");
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2627 assert(isValidChar(d), "check if character is valid before calling encodeUTF8().");
430
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 426
diff changeset
2628
519
9ebc799c7dc5 Fixes in dil.Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 514
diff changeset
2629 char[6] b = void;
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2630 if (d < 0x800)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2631 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2632 b[0] = 0xC0 | (d >> 6);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2633 b[1] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2634 str ~= b[0..2];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2635 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2636 else if (d < 0x10000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2637 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2638 b[0] = 0xE0 | (d >> 12);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2639 b[1] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2640 b[2] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2641 str ~= b[0..3];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2642 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2643 else if (d < 0x200000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2644 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2645 b[0] = 0xF0 | (d >> 18);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2646 b[1] = 0x80 | ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2647 b[2] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2648 b[3] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2649 str ~= b[0..4];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2650 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2651 /+ // There are no 5 and 6 byte UTF-8 sequences yet.
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2652 else if (d < 0x4000000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2653 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2654 b[0] = 0xF8 | (d >> 24);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2655 b[1] = 0x80 | ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2656 b[2] = 0x80 | ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2657 b[3] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2658 b[4] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2659 str ~= b[0..5];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2660 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2661 else if (d < 0x80000000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2662 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2663 b[0] = 0xFC | (d >> 30);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2664 b[1] = 0x80 | ((d >> 24) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2665 b[2] = 0x80 | ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2666 b[3] = 0x80 | ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2667 b[4] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2668 b[5] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2669 str ~= b[0..6];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2670 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2671 +/
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2672 else
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2673 assert(0);
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2674 }
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
2675 }
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2676
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2677 unittest
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2678 {
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
2679 Stdout("Testing Lexer.\n");
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2680 struct Pair
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2681 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2682 char[] tokenText;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2683 TOK kind;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2684 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2685 static Pair[] pairs = [
494
9a7ca8c56e59 Refactored a few things in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
2686 {"#!äöüß", TOK.Shebang}, {"\n", TOK.Newline},
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2687 {"//çay", TOK.Comment}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2688 {"&", TOK.AndBinary},
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2689 {"/*çağ*/", TOK.Comment}, {"&&", TOK.AndLogical},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2690 {"/+çak+/", TOK.Comment}, {"&=", TOK.AndAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2691 {">", TOK.Greater}, {"+", TOK.Plus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2692 {">=", TOK.GreaterEqual}, {"++", TOK.PlusPlus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2693 {">>", TOK.RShift}, {"+=", TOK.PlusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2694 {">>=", TOK.RShiftAssign}, {"-", TOK.Minus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2695 {">>>", TOK.URShift}, {"--", TOK.MinusMinus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2696 {">>>=", TOK.URShiftAssign}, {"-=", TOK.MinusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2697 {"<", TOK.Less}, {"=", TOK.Assign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2698 {"<=", TOK.LessEqual}, {"==", TOK.Equal},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2699 {"<>", TOK.LorG}, {"~", TOK.Tilde},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2700 {"<>=", TOK.LorEorG}, {"~=", TOK.CatAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2701 {"<<", TOK.LShift}, {"*", TOK.Mul},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2702 {"<<=", TOK.LShiftAssign}, {"*=", TOK.MulAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2703 {"!", TOK.Not}, {"/", TOK.Div},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2704 {"!=", TOK.NotEqual}, {"/=", TOK.DivAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2705 {"!<", TOK.UorGorE}, {"^", TOK.Xor},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2706 {"!>", TOK.UorLorE}, {"^=", TOK.XorAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2707 {"!<=", TOK.UorG}, {"%", TOK.Mod},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2708 {"!>=", TOK.UorL}, {"%=", TOK.ModAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2709 {"!<>", TOK.UorE}, {"(", TOK.LParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2710 {"!<>=", TOK.Unordered}, {")", TOK.RParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2711 {".", TOK.Dot}, {"[", TOK.LBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2712 {"..", TOK.Slice}, {"]", TOK.RBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2713 {"...", TOK.Ellipses}, {"{", TOK.LBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2714 {"|", TOK.OrBinary}, {"}", TOK.RBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2715 {"||", TOK.OrLogical}, {":", TOK.Colon},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2716 {"|=", TOK.OrAssign}, {";", TOK.Semicolon},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2717 {"?", TOK.Question}, {",", TOK.Comma},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2718 {"$", TOK.Dollar}, {"cam", TOK.Identifier},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2719 {"çay", TOK.Identifier}, {".0", TOK.Float64},
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2720 {"0", TOK.Int32}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2721 {"\r", TOK.Newline}, {"\r\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2722 {"\u2028", TOK.Newline}, {"\u2029", TOK.Newline}
40
9d5ceb0f8be9 - Added more tokens for testing.
aziz
parents: 39
diff changeset
2723 ];
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2724
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2725 char[] src;
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2726
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2727 // Join all token texts into a single string.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2728 foreach (i, pair; pairs)
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2729 if (pair.kind == TOK.Comment && pair.tokenText[1] == '/' || // Line comment.
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2730 pair.kind == TOK.Shebang)
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2731 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2732 assert(pairs[i+1].kind == TOK.Newline); // Must be followed by a newline.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2733 src ~= pair.tokenText;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2734 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2735 else
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2736 src ~= pair.tokenText ~ " ";
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2737
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2738 auto lx = new Lexer(src, "");
350
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2739 auto token = lx.getTokens();
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2740
350
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2741 uint i;
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2742 assert(token == lx.head);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2743 assert(token.next.kind == TOK.Newline);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2744 token = token.next.next;
350
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2745 do
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2746 {
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2747 assert(i < pairs.length);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2748 assert(token.srcText == pairs[i].tokenText, Format("Scanned '{0}' but expected '{1}'", token.srcText, pairs[i].tokenText));
350
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2749 ++i;
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2750 token = token.next;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2751 } while (token.kind != TOK.EOF)
41
2b7be1d67d4d - Optimized scanner of block comments.
aziz
parents: 40
diff changeset
2752 }
55
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2753
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2754 unittest
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2755 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2756 Stdout("Testing method Lexer.peek()\n");
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2757 string sourceText = "unittest { }";
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2758 auto lx = new Lexer(sourceText, null);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2759
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2760 auto next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2761 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2762 assert(next.kind == TOK.Newline);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2763 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2764 assert(next.kind == TOK.Unittest);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2765 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2766 assert(next.kind == TOK.LBrace);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2767 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2768 assert(next.kind == TOK.RBrace);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2769 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2770 assert(next.kind == TOK.EOF);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2771
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2772 lx = new Lexer("", null);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2773 next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2774 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2775 assert(next.kind == TOK.Newline);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2776 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2777 assert(next.kind == TOK.EOF);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2778 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2779
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2780 unittest
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2781 {
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2782 // Numbers unittest
68
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2783 // 0L 0ULi 0_L 0_UL 0x0U 0x0p2 0_Fi 0_e2 0_F 0_i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2784 // 0u 0U 0uL 0UL 0L 0LU 0Lu
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2785 // 0Li 0f 0F 0fi 0Fi 0i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2786 // 0b_1_LU 0b1000u
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2787 // 0x232Lu
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2788 }