Mercurial > projects > dil

annotate trunk/src/dil/Lexer.d @ 552:3bc7801c207e

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Refactored the way how tokens are flagged as whitespace.
author Aziz K?ksal <aziz.koeksal@gmail.com>
date Thu, 20 Dec 2007 23:26:43 +0100
parents 082126d78f90
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
1 /++
8ba2570de175 Initial import.
aziz
parents:
diff changeset
2 Author: Aziz Köksal
249
32d354584b28 - Upgraded license notices to GPL3.
aziz
parents: 239
diff changeset
3 License: GPL3
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
4 +/
326
4a7359b88c11 - Added package 'dil' to module declarations.
aziz
parents: 325
diff changeset
5 module dil.Lexer;
327
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
6 import dil.Token;
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
7 import dil.Information;
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
8 import dil.Keywords;
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
9 import dil.Identifier;
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
10 import dil.Messages;
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
11 import dil.HtmlEntities;
464
325714d8aa6c Added new module with compiler constants to resolve import cycle.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 434
diff changeset
12 import dil.CompilerInfo;
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
13 import dil.IdTable;
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
14 import dil.Unicode;
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
15 import tango.stdc.stdlib : strtof, strtod, strtold;
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
16 import tango.stdc.errno : errno, ERANGE;
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
17 import tango.stdc.time : time_t, time, ctime;
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
18 import tango.stdc.string : strlen;
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
19 import common;
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
20
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 489
diff changeset
21 public import dil.LexerFuncs;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
22
499
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
23 /++
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
24 The Lexer analyzes the characters of a source text and
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
25 produces a doubly-linked list of tokens.
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
26 +/
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
27 class Lexer
8ba2570de175 Initial import.
aziz
parents:
diff changeset
28 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
29 Token* head; /// The head of the doubly linked token list.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
30 Token* tail; /// The tail of the linked list. Set in scan().
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
31 Token* token; /// Points to the current token in the token list.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
32 string text; /// The source text.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
33 char[] filePath; /// Path to the source text.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
34 char* p; /// Points to the current character in the source text.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
35 char* end; /// Points one character past the end of the source text.
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
36
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
37 // Members used for error messages:
532
50e64bab9c7a Renamed InformationManager to InfoManager.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 520
diff changeset
38 InfoManager infoMan;
514
6ddff941862a Added new error classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 510
diff changeset
39 LexerError[] errors;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
40 /// Always points to the beginning of the current line.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
41 char* lineBegin;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
42 // Token* newline; /// Current newline token.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
43 uint lineNum = 1; /// Current, actual source text line number.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
44 uint lineNum_hline; /// Line number set by #line.
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
45 uint inTokenString; /// > 0 if inside q{ }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
46 char[] errorPath; /// The path displayed in error messages.
103
511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too.
aziz
parents: 94
diff changeset
47
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
48 /++
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
49 Construct a Lexer object.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
50 Params:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
51 text = the UTF-8 source code.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
52 filePath = the path to the source code; used for error messages.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
53 +/
532
50e64bab9c7a Renamed InformationManager to InfoManager.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 520
diff changeset
54 this(string text, string filePath, InfoManager infoMan = null)
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
55 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
56 this.filePath = this.errorPath = filePath;
520
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
57 this.infoMan = infoMan;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
58
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
59 this.text = text;
397
c99f8aeb7b4a Empty source files are handled correctly now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 396
diff changeset
60 if (text.length == 0 || text[$-1] != 0)
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
61 {
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
62 this.text.length = this.text.length + 1;
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
63 this.text[$-1] = 0;
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
64 }
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
65
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
66 this.p = this.text.ptr;
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
67 this.end = this.p + this.text.length;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
68 this.lineBegin = this.p;
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
69
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
70 this.head = new Token;
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
71 this.head.type = TOK.HEAD;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
72 this.head.start = this.head.end = this.p;
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
73 this.token = this.head;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
74 // Add a newline as the first token after the head.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
75 auto newline = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
76 newline.type = TOK.Newline;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
77 newline.setWhitespaceFlag();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
78 newline.start = newline.end = this.p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
79 newline.filePath = this.errorPath;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
80 newline.lineNum = 1;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
81 newline.lineNum_hline = 0;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
82 // Link in.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
83 this.token.next = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
84 newline.prev = this.token;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
85 this.token = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
86 // this.newline = newline;
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
87 scanShebang();
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
88 }
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
89
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
90 ~this()
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
91 {
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
92 auto token = head.next;
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
93 while (token !is null)
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
94 {
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
95 assert(token.type == TOK.EOF ? token == tail && token.next is null : 1);
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
96 delete token.prev;
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
97 token = token.next;
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
98 }
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
99 delete tail;
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
100 }
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
101
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
102 /++
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
103 The "shebang" may optionally appear once at the beginning of a file.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
104 Regexp: #![^\EndOfLine]*
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
105 +/
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
106 void scanShebang()
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
107 {
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
108 if (*p == '#' && p[1] == '!')
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
109 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
110 auto t = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
111 t.type = TOK.Shebang;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
112 t.setWhitespaceFlag();
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
113 t.start = p;
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
114 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
115 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
116 isascii(*p) || decodeUTF8();
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
117 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
118 this.token.next = t;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
119 t.prev = this.token;
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
120 }
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
121 }
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
122
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
123 void finalizeSpecialToken(ref Token t)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
124 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
125 assert(t.srcText[0..2] == "__");
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
126 switch (t.type)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
127 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
128 case TOK.FILE:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
129 t.str = this.errorPath;
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
130 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
131 case TOK.LINE:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
132 t.uint_ = this.errorLineNumber(this.lineNum);
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
133 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
134 case TOK.DATE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
135 TOK.TIME,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
136 TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
137 time_t time_val;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
138 time(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
139 char* str = ctime(&time_val);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
140 char[] time_str = str[0 .. strlen(str)];
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
141 switch (t.type)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
142 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
143 case TOK.DATE:
346
ce9e2e77743f - Fix: include one space when slicing time_str for __DATE__.
aziz
parents: 344
diff changeset
144 time_str = time_str[4..11] ~ time_str[20..24] ~ \0; break;
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
145 case TOK.TIME:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
146 time_str = time_str[11..19] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
147 case TOK.TIMESTAMP:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
148 time_str = time_str[0..24] ~ \0; break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
149 default: assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
150 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
151 t.str = time_str;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
152 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
153 case TOK.VENDOR:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
154 t.str = VENDOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
155 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
156 case TOK.VERSION:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
157 t.uint_ = VERSION_MAJOR*1000 + VERSION_MINOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
158 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
159 default:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
160 assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
161 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
162 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
163
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
164 private void setLineBegin(char* p)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
165 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
166 // Check that we can look behind one character.
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
167 assert((p-1) >= text.ptr && p < end);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
168 // Check that previous character is a newline.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
169 assert(isNewlineEnd(p - 1));
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
170 this.lineBegin = p;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
171 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
172
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
173 private void scanNext(ref Token* t)
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
174 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
175 assert(t !is null);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
176 if (t.next)
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
177 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
178 t = t.next;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
179 // if (t.type == TOK.Newline)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
180 // this.newline = t;
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
181 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
182 else if (t != this.tail)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
183 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
184 Token* new_t = new Token;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
185 scan(*new_t);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
186 new_t.prev = t;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
187 t.next = new_t;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
188 t = new_t;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
189 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
190 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
191
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
192 /// Advance t one token forward.
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
193 void peek(ref Token* t)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
194 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
195 scanNext(t);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
196 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
197
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
198 /// Advance to the next token in the source text.
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
199 TOK nextToken()
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
200 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
201 scanNext(this.token);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
202 return this.token.type;
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
203 }
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
204
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
205 /// Returns true if p points to the last character of a Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
206 bool isNewlineEnd(char* p)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
207 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
208 if (*p == '\n' || *p == '\r')
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
209 return true;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
210 if (*p == LS[2] || *p == PS[2])
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
211 if ((p-2) >= text.ptr)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
212 if (p[-1] == LS[1] && p[-2] == LS[0])
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
213 return true;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
214 return false;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
215 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
216
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
217 /++
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
218 This is the old scan method.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
219 TODO: profile old and new to see which one is faster.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
220 +/
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
221 public void scan(ref Token t)
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
222 in
3
4bbce78bfb1e - Added TOK enum.
aziz
parents: 2
diff changeset
223 {
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
224 assert(text.ptr <= p && p < end);
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
225 }
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
226 out
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
227 {
397
c99f8aeb7b4a Empty source files are handled correctly now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 396
diff changeset
228 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type));
c99f8aeb7b4a Empty source files are handled correctly now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 396
diff changeset
229 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type));
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
230 }
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
231 body
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
232 {
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
233 // Scan whitespace.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
234 if (isspace(*p))
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
235 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
236 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
237 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
238 {}
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
239 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
240
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
241 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
242 uint c = *p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
243 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
244 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
245 // Newline.
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
246 switch (*p)
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
247 {
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
248 case '\r':
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
249 if (p[1] == '\n')
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
250 ++p;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
251 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
252 assert(isNewlineEnd(p));
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
253 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
254 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
255 setLineBegin(p);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
256 // this.newline = &t;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
257 t.type = TOK.Newline;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
258 t.setWhitespaceFlag();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
259 t.filePath = this.errorPath;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
260 t.lineNum = lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
261 t.lineNum_hline = lineNum_hline;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
262 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
263 return;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
264 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
265 if (isUnicodeNewline(p))
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
266 {
412
fb31af0fda73 Added struct Location, and token2LocTable to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 411
diff changeset
267 ++p; ++p;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 411
diff changeset
268 goto case '\n';
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
269 }
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
270 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
271 // Identifier or string literal.
13
e5211758b63c - Added isidbeg() function.
aziz
parents: 12
diff changeset
272 if (isidbeg(c))
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
273 {
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
274 if (c == 'r' && p[1] == '"' && ++p)
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
275 return scanRawStringLiteral(t);
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
276 if (c == 'x' && p[1] == '"')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
277 return scanHexStringLiteral(t);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
278 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
279 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
280 if (c == 'q' && p[1] == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
281 return scanDelimitedStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
282 if (c == 'q' && p[1] == '{')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
283 return scanTokenStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
284 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
285 // Scan identifier.
12
0989206cf73c - Added code to decode Unicode characters in identifiers.
aziz
parents: 11
diff changeset
286 Lidentifier:
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
287 do
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
288 { c = *++p; }
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
289 while (isident(c) || !isascii(c) && isUnicodeAlpha())
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
290
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
291 t.end = p;
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
292
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
293 auto id = IdTable.lookup(t.srcText);
499
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
294 t.type = id.type;
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
295 t.ident = id;
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
296
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
297 if (t.type == TOK.Identifier || t.isKeyword)
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
298 return;
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
299 else if (t.isSpecialToken)
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
300 finalizeSpecialToken(t);
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
301 else if (t.type == TOK.EOF)
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
302 {
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
303 tail = &t;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
304 assert(t.srcText == "__EOF__");
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
305 }
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
306 else
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
307 assert(0, "unexpected token type: " ~ Token.toString(t.type));
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
308 return;
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
309 }
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
310
15
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
311 if (isdigit(c))
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
312 return scanNumber(t);
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
313
8
d4ba94a5a282 - Parsing /* */ comments now.
aziz
parents: 7
diff changeset
314 if (c == '/')
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
315 {
8
d4ba94a5a282 - Parsing /* */ comments now.
aziz
parents: 7
diff changeset
316 c = *++p;
14
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
317 switch(c)
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
318 {
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
319 case '=':
14
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
320 ++p;
32
d7011daa4740 - Added missing commas after the items in the messages table.
aziz
parents: 31
diff changeset
321 t.type = TOK.DivAssign;
14
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
322 t.end = p;
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
323 return;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
324 case '+':
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
325 return scanNestedComment(t);
32
d7011daa4740 - Added missing commas after the items in the messages table.
aziz
parents: 31
diff changeset
326 case '*':
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
327 return scanBlockComment(t);
32
d7011daa4740 - Added missing commas after the items in the messages table.
aziz
parents: 31
diff changeset
328 case '/':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
329 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
330 isascii(*p) || decodeUTF8();
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
331 t.type = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
332 t.setWhitespaceFlag();
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
333 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
334 return;
32
d7011daa4740 - Added missing commas after the items in the messages table.
aziz
parents: 31
diff changeset
335 default:
d7011daa4740 - Added missing commas after the items in the messages table.
aziz
parents: 31
diff changeset
336 t.type = TOK.Div;
10
3ee65d6e39c9 - Parsing // comments now.
aziz
parents: 9
diff changeset
337 t.end = p;
3ee65d6e39c9 - Parsing // comments now.
aziz
parents: 9
diff changeset
338 return;
3ee65d6e39c9 - Parsing // comments now.
aziz
parents: 9
diff changeset
339 }
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
340 }
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
341
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
342 switch (c)
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
343 {
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
344 case '\'':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
345 return scanCharacterLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
346 case '`':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
347 return scanRawStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
348 case '"':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
349 return scanNormalStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
350 case '\\':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
351 char[] buffer;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
352 do
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
353 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
354 c = scanEscapeSequence();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
355 if (isascii(c))
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
356 buffer ~= c;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
357 else
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
358 encodeUTF8(buffer, c);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
359 } while (*p == '\\')
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
360 buffer ~= 0;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
361 t.type = TOK.String;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
362 t.str = buffer;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
363 t.end = p;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
364 return;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
365 case '>': /* > >= >> >>= >>> >>>= */
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
366 c = *++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
367 switch (c)
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
368 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
369 case '=':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
370 t.type = TOK.GreaterEqual;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
371 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
372 case '>':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
373 if (p[1] == '>')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
374 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
375 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
376 if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
377 { ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
378 t.type = TOK.URShiftAssign;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
379 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
380 else
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
381 t.type = TOK.URShift;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
382 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
383 else if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
384 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
385 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
386 t.type = TOK.RShiftAssign;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
387 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
388 else
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
389 t.type = TOK.RShift;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
390 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
391 default:
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
392 t.type = TOK.Greater;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
393 goto Lcommon2;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
394 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
395 assert(0);
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
396 case '<': /* < <= <> <>= << <<= */
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
397 c = *++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
398 switch (c)
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
399 {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
400 case '=':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
401 t.type = TOK.LessEqual;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
402 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
403 case '<':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
404 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
405 ++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
406 t.type = TOK.LShiftAssign;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
407 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
408 else
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
409 t.type = TOK.LShift;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
410 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
411 case '>':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
412 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
413 ++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
414 t.type = TOK.LorEorG;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
415 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
416 else
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
417 t.type = TOK.LorG;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
418 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
419 default:
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
420 t.type = TOK.Less;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
421 goto Lcommon2;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
422 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
423 assert(0);
37
7f3bcb97d017 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 36
diff changeset
424 case '!': /* ! !< !> !<= !>= !<> !<>= */
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
425 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
426 switch (c)
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
427 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
428 case '<':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
429 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
430 if (c == '>')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
431 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
432 if (p[1] == '=') {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
433 ++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
434 t.type = TOK.Unordered;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
435 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
436 else
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
437 t.type = TOK.UorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
438 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
439 else if (c == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
440 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
441 t.type = TOK.UorG;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
442 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
443 else {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
444 t.type = TOK.UorGorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
445 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
446 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
447 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
448 case '>':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
449 if (p[1] == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
450 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
451 ++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
452 t.type = TOK.UorL;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
453 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
454 else
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
455 t.type = TOK.UorLorE;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
456 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
457 case '=':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
458 t.type = TOK.NotEqual;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
459 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
460 default:
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
461 t.type = TOK.Not;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
462 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
463 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
464 assert(0);
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
465 case '.': /* . .[0-9] .. ... */
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
466 if (p[1] == '.')
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
467 {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
468 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
469 if (p[1] == '.') {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
470 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
471 t.type = TOK.Ellipses;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
472 }
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
473 else
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
474 t.type = TOK.Slice;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
475 }
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
476 else if (isdigit(p[1]))
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
477 {
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
478 return scanReal(t);
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
479 }
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
480 else
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
481 t.type = TOK.Dot;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
482 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
483 case '|': /* | || |= */
23
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
484 c = *++p;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
485 if (c == '=')
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
486 t.type = TOK.OrAssign;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
487 else if (c == '|')
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
488 t.type = TOK.OrLogical;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
489 else {
23
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
490 t.type = TOK.OrBinary;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
491 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
492 }
23
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
493 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
494 case '&': /* & && &= */
24
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
495 c = *++p;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
496 if (c == '=')
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
497 t.type = TOK.AndAssign;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
498 else if (c == '&')
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
499 t.type = TOK.AndLogical;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
500 else {
24
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
501 t.type = TOK.AndBinary;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
502 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
503 }
24
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
504 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
505 case '+': /* + ++ += */
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
506 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
507 if (c == '=')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
508 t.type = TOK.PlusAssign;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
509 else if (c == '+')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
510 t.type = TOK.PlusPlus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
511 else {
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
512 t.type = TOK.Plus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
513 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
514 }
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
515 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
516 case '-': /* - -- -= */
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
517 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
518 if (c == '=')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
519 t.type = TOK.MinusAssign;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
520 else if (c == '-')
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
521 t.type = TOK.MinusMinus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
522 else {
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
523 t.type = TOK.Minus;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
524 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
525 }
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
526 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
527 case '=': /* = == */
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
528 if (p[1] == '=') {
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
529 ++p;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
530 t.type = TOK.Equal;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
531 }
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
532 else
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
533 t.type = TOK.Assign;
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
534 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
535 case '~': /* ~ ~= */
27
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
536 if (p[1] == '=') {
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
537 ++p;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
538 t.type = TOK.CatAssign;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
539 }
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
540 else
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
541 t.type = TOK.Tilde;
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
542 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
543 case '*': /* * *= */
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
544 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
545 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
546 t.type = TOK.MulAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
547 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
548 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
549 t.type = TOK.Mul;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
550 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
551 case '^': /* ^ ^= */
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
552 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
553 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
554 t.type = TOK.XorAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
555 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
556 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
557 t.type = TOK.Xor;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
558 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
559 case '%': /* % %= */
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
560 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
561 ++p;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
562 t.type = TOK.ModAssign;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
563 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
564 else
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
565 t.type = TOK.Mod;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
566 goto Lcommon;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
567 // Single character tokens:
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
568 case '(':
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
569 t.type = TOK.LParen;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
570 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
571 case ')':
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
572 t.type = TOK.RParen;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
573 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
574 case '[':
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
575 t.type = TOK.LBracket;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
576 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
577 case ']':
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
578 t.type = TOK.RBracket;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
579 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
580 case '{':
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
581 t.type = TOK.LBrace;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
582 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
583 case '}':
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
584 t.type = TOK.RBrace;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
585 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
586 case ':':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
587 t.type = TOK.Colon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
588 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
589 case ';':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
590 t.type = TOK.Semicolon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
591 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
592 case '?':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
593 t.type = TOK.Question;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
594 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
595 case ',':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
596 t.type = TOK.Comma;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
597 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
598 case '$':
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
599 t.type = TOK.Dollar;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
600 Lcommon:
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
601 ++p;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
602 Lcommon2:
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
603 t.end = p;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
604 return;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
605 case '#':
360
b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence().
aziz
parents: 350
diff changeset
606 return scanSpecialTokenSequence(t);
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
607 default:
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
608 }
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
609
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
610 // Check for EOF
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
611 if (isEOF(c))
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
612 {
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
613 assert(isEOF(*p), ""~*p);
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
614 t.type = TOK.EOF;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
615 t.end = p;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
616 tail = &t;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
617 assert(t.start == t.end);
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
618 return;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
619 }
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
620
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
621 if (!isascii(c))
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
622 {
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
623 c = decodeUTF8();
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
624 if (isUniAlpha(c))
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
625 goto Lidentifier;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
626 }
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
627
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
628 error(t.start, MID.IllegalCharacter, cast(dchar)c);
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
629
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
630 ++p;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
631 t.type = TOK.Illegal;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
632 t.setWhitespaceFlag();
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
633 t.dchar_ = c;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
634 t.end = p;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
635 return;
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
636 }
3
4bbce78bfb1e - Added TOK enum.
aziz
parents: 2
diff changeset
637 }
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
638
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
639 template toUint(char[] T)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
640 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
641 static assert(0 < T.length && T.length <= 4);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
642 static if (T.length == 1)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
643 const uint toUint = T[0];
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
644 else
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
645 const uint toUint = (T[0] << ((T.length-1)*8)) | toUint!(T[1..$]);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
646 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
647 static assert(toUint!("\xAA\xBB\xCC\xDD") == 0xAABBCCDD);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
648
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
649 // Can't use this yet due to a bug in DMD (bug id=1534).
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
650 template case_(char[] str, TOK tok, char[] label)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
651 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
652 const char[] case_ =
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
653 `case `~toUint!(str).stringof~`:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
654
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
655 goto `~label~`;`;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
656 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
657
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
658 template case_L4(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
659 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
660 const char[] case_L4 = case_!(str, tok, "Lcommon_4");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
661 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
662
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
663 template case_L3(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
664 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
665 const char[] case_L3 = case_!(str, tok, "Lcommon_3");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
666 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
667
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
668 template case_L2(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
669 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
670 const char[] case_L2 = case_!(str, tok, "Lcommon_2");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
671 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
672
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
673 template case_L1(char[] str, TOK tok)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
674 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
675 const char[] case_L3 = case_!(str, tok, "Lcommon");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
676 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
677
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
678 public void scan_(ref Token t)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
679 in
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
680 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
681 assert(text.ptr <= p && p < end);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
682 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
683 out
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
684 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
685 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type));
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
686 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type));
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
687 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
688 body
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
689 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
690 // Scan whitespace.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
691 if (isspace(*p))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
692 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
693 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
694 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
695 {}
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
696 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
697
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
698 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
699 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
700 // Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
701 switch (*p)
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
702 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
703 case '\r':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
704 if (p[1] == '\n')
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
705 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
706 case '\n':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
707 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
708 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
709 ++lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
710 setLineBegin(p);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
711 // this.newline = &t;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
712 t.type = TOK.Newline;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
713 t.setWhitespaceFlag();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
714 t.filePath = this.errorPath;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
715 t.lineNum = lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
716 t.lineNum_hline = lineNum_hline;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
717 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
718 return;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
719 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
720 if (isUnicodeNewline(p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
721 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
722 ++p; ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
723 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
724 }
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
725 }
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
726
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
727 uint c = *p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
728 assert(end - p != 0);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
729 switch (end - p)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
730 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
731 case 1:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
732 goto L1character;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
733 case 2:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
734 c <<= 8; c |= p[1];
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
735 goto L2characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
736 case 3:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
737 c <<= 8; c |= p[1]; c <<= 8; c |= p[2];
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
738 goto L3characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
739 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
740 version(BigEndian)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
741 c = *cast(uint*)p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
742 else
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
743 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
744 c <<= 8; c |= p[1]; c <<= 8; c |= p[2]; c <<= 8; c |= p[3];
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
745 /+
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
746 c = *cast(uint*)p;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
747 asm
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
748 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
749 mov EDX, c;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
750 bswap EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
751 mov c, EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
752 }
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
753 +/
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
754 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
755 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
756
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
757 // 4 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
758 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
759 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
760 case toUint!(">>>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
761 t.type = TOK.RShiftAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
762 goto Lcommon_4;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
763 case toUint!("!<>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
764 t.type = TOK.Unordered;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
765 Lcommon_4:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
766 p += 4;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
767 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
768 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
769 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
770 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
771
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
772 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
773 L3characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
774 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
775 // 3 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
776 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
777 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
778 case toUint!(">>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
779 t.type = TOK.RShiftAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
780 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
781 case toUint!(">>>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
782 t.type = TOK.URShift;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
783 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
784 case toUint!("<>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
785 t.type = TOK.LorEorG;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
786 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
787 case toUint!("<<="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
788 t.type = TOK.LShiftAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
789 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
790 case toUint!("!<="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
791 t.type = TOK.UorG;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
792 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
793 case toUint!("!>="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
794 t.type = TOK.UorL;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
795 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
796 case toUint!("!<>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
797 t.type = TOK.UorE;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
798 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
799 case toUint!("..."):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
800 t.type = TOK.Ellipses;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
801 Lcommon_3:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
802 p += 3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
803 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
804 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
805 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
806 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
807
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
808 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
809 L2characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
810 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
811 // 2 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
812 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
813 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
814 case toUint!("/+"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
815 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
816 return scanNestedComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
817 case toUint!("/*"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
818 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
819 return scanBlockComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
820 case toUint!("//"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
821 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
822 assert(*p == '/');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
823 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
824 isascii(*p) || decodeUTF8();
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
825 t.type = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
826 t.setWhitespaceFlag();
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
827 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
828 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
829 case toUint!(">="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
830 t.type = TOK.GreaterEqual;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
831 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
832 case toUint!(">>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
833 t.type = TOK.RShift;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
834 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
835 case toUint!("<<"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
836 t.type = TOK.LShift;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
837 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
838 case toUint!("<="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
839 t.type = TOK.LessEqual;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
840 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
841 case toUint!("<>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
842 t.type = TOK.LorG;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
843 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
844 case toUint!("!<"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
845 t.type = TOK.UorGorE;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
846 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
847 case toUint!("!>"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
848 t.type = TOK.UorLorE;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
849 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
850 case toUint!("!="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
851 t.type = TOK.NotEqual;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
852 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
853 case toUint!(".."):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
854 t.type = TOK.Slice;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
855 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
856 case toUint!("&&"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
857 t.type = TOK.AndLogical;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
858 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
859 case toUint!("&="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
860 t.type = TOK.AndAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
861 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
862 case toUint!("||"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
863 t.type = TOK.OrLogical;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
864 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
865 case toUint!("|="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
866 t.type = TOK.OrAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
867 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
868 case toUint!("++"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
869 t.type = TOK.PlusPlus;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
870 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
871 case toUint!("+="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
872 t.type = TOK.PlusAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
873 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
874 case toUint!("--"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
875 t.type = TOK.MinusMinus;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
876 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
877 case toUint!("-="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
878 t.type = TOK.MinusAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
879 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
880 case toUint!("=="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
881 t.type = TOK.Equal;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
882 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
883 case toUint!("~="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
884 t.type = TOK.CatAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
885 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
886 case toUint!("*="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
887 t.type = TOK.MulAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
888 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
889 case toUint!("/="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
890 t.type = TOK.DivAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
891 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
892 case toUint!("^="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
893 t.type = TOK.XorAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
894 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
895 case toUint!("%="):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
896 t.type = TOK.ModAssign;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
897 Lcommon_2:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
898 p += 2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
899 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
900 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
901 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
902 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
903
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
904 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
905 L1character:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
906 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
907 assert(*p == c, Format("p={0},c={1}", *p, cast(dchar)c));
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
908 // 1 character tokens.
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
909 // TODO: consider storing the token type in ptable.
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
910 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
911 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
912 case '\'':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
913 return scanCharacterLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
914 case '`':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
915 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
916 case '"':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
917 return scanNormalStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
918 case '\\':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
919 char[] buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
920 do
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
921 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
922 c = scanEscapeSequence();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
923 if (isascii(c))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
924 buffer ~= c;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
925 else
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
926 encodeUTF8(buffer, c);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
927 } while (*p == '\\')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
928 buffer ~= 0;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
929 t.type = TOK.String;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
930 t.str = buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
931 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
932 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
933 case '<':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
934 t.type = TOK.Greater;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
935 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
936 case '>':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
937 t.type = TOK.Less;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
938 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
939 case '^':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
940 t.type = TOK.Xor;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
941 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
942 case '!':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
943 t.type = TOK.Not;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
944 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
945 case '.':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
946 if (isdigit(p[1]))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
947 return scanReal(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
948 t.type = TOK.Dot;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
949 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
950 case '&':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
951 t.type = TOK.AndBinary;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
952 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
953 case '|':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
954 t.type = TOK.OrBinary;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
955 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
956 case '+':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
957 t.type = TOK.Plus;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
958 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
959 case '-':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
960 t.type = TOK.Minus;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
961 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
962 case '=':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
963 t.type = TOK.Assign;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
964 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
965 case '~':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
966 t.type = TOK.Tilde;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
967 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
968 case '*':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
969 t.type = TOK.Mul;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
970 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
971 case '/':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
972 t.type = TOK.Div;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
973 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
974 case '%':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
975 t.type = TOK.Mod;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
976 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
977 case '(':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
978 t.type = TOK.LParen;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
979 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
980 case ')':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
981 t.type = TOK.RParen;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
982 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
983 case '[':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
984 t.type = TOK.LBracket;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
985 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
986 case ']':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
987 t.type = TOK.RBracket;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
988 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
989 case '{':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
990 t.type = TOK.LBrace;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
991 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
992 case '}':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
993 t.type = TOK.RBrace;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
994 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
995 case ':':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
996 t.type = TOK.Colon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
997 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
998 case ';':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
999 t.type = TOK.Semicolon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1000 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1001 case '?':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1002 t.type = TOK.Question;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1003 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1004 case ',':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1005 t.type = TOK.Comma;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1006 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1007 case '$':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1008 t.type = TOK.Dollar;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1009 Lcommon:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1010 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1011 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1012 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1013 case '#':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1014 return scanSpecialTokenSequence(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1015 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1016 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1017
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1018 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1019 assert(*p == c);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1020
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1021 // TODO: consider moving isidbeg() and isdigit() up.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1022 if (isidbeg(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1023 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1024 if (c == 'r' && p[1] == '"' && ++p)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1025 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1026 if (c == 'x' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1027 return scanHexStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1028 version(D2)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1029 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1030 if (c == 'q' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1031 return scanDelimitedStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1032 if (c == 'q' && p[1] == '{')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1033 return scanTokenStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1034 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1035 // Scan identifier.
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1036 Lidentifier:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1037 do
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1038 { c = *++p; }
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
1039 while (isident(c) || !isascii(c) && isUnicodeAlpha())
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1040
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1041 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1042
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
1043 auto id = IdTable.lookup(t.srcText);
499
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
1044 t.type = id.type;
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
1045 t.ident = id;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1046
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1047 if (t.type == TOK.Identifier || t.isKeyword)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1048 return;
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1049 else if (t.isSpecialToken)
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1050 finalizeSpecialToken(t);
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1051 else if (t.type == TOK.EOF)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1052 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1053 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1054 assert(t.srcText == "__EOF__");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1055 }
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1056 else
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
1057 assert(0, "unexpected token type: " ~ Token.toString(t.type));
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1058 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1059 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1060
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1061 if (isdigit(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1062 return scanNumber(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1063
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1064 // Check for EOF
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1065 if (isEOF(c))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1066 {
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1067 assert(isEOF(*p), *p~"");
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1068 t.type = TOK.EOF;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1069 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1070 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1071 assert(t.start == t.end);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1072 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1073 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1074
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1075 if (!isascii(c))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1076 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1077 c = decodeUTF8();
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1078 if (isUniAlpha(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1079 goto Lidentifier;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1080 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1081
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1082 error(t.start, MID.IllegalCharacter, cast(dchar)c);
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1083
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1084 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1085 t.type = TOK.Illegal;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
1086 t.setWhitespaceFlag();
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1087 t.dchar_ = c;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1088 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1089 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1090 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1091
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1092 void scanBlockComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1093 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1094 assert(p[-1] == '/' && *p == '*');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1095 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1096 auto tokenLineBegin = lineBegin;
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1097 Loop:
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1098 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1099 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1100 switch (*++p)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1101 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1102 case '*':
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1103 if (p[1] != '/')
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1104 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1105 p += 2;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1106 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1107 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1108 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1109 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1110 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1111 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1112 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1113 setLineBegin(p+1);
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1114 break;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1115 default:
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1116 if (!isascii(*p))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1117 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1118 if (isUnicodeNewlineChar(decodeUTF8()))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1119 goto case '\n';
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1120 }
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1121 else if (isEOF(*p))
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1122 {
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1123 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedBlockComment);
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1124 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1125 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1126 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1127 }
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1128 t.type = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
1129 t.setWhitespaceFlag();
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1130 t.end = p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1131 return;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1132 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1133
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1134 void scanNestedComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1135 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1136 assert(p[-1] == '/' && *p == '+');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1137 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1138 auto tokenLineBegin = lineBegin;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1139 uint level = 1;
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1140 Loop:
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1141 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1142 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1143 switch (*++p)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1144 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1145 case '/':
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1146 if (p[1] == '+')
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1147 ++p, ++level;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1148 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1149 case '+':
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1150 if (p[1] != '/')
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1151 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1152 ++p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1153 if (--level != 0)
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1154 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1155 ++p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1156 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1157 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1158 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1159 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1160 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1161 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1162 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1163 setLineBegin(p+1);
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1164 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1165 default:
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1166 if (!isascii(*p))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1167 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1168 if (isUnicodeNewlineChar(decodeUTF8()))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1169 goto case '\n';
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1170 }
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1171 else if (isEOF(*p))
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1172 {
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1173 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedNestedComment);
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1174 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1175 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1176 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1177 }
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1178 t.type = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
1179 t.setWhitespaceFlag();
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1180 t.end = p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1181 return;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1182 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1183
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1184 char scanPostfix()
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1185 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1186 assert(p[-1] == '"' || p[-1] == '`' ||
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1187 { version(D2) return p[-1] == '}';
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1188 else return 0; }()
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1189 );
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1190 switch (*p)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1191 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1192 case 'c':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1193 case 'w':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1194 case 'd':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1195 return *p++;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1196 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1197 return 0;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1198 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1199 assert(0);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1200 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1201
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1202 void scanNormalStringLiteral(ref Token t)
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1203 {
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1204 assert(*p == '"');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1205 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1206 auto tokenLineBegin = lineBegin;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1207 t.type = TOK.String;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1208 char[] buffer;
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1209 uint c;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1210 while (1)
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1211 {
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1212 c = *++p;
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1213 switch (c)
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1214 {
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1215 case '"':
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1216 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1217 t.pf = scanPostfix();
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1218 Lreturn:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1219 t.str = buffer ~ '\0';
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1220 t.end = p;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1221 return;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1222 case '\\':
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1223 c = scanEscapeSequence();
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1224 --p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1225 if (isascii(c))
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1226 break;
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1227 encodeUTF8(buffer, c);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1228 continue;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1229 case '\r':
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1230 if (p[1] == '\n')
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1231 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1232 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1233 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1234 c = '\n'; // Convert Newline to \n.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1235 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1236 setLineBegin(p+1);
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1237 break;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1238 case 0, _Z_:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1239 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedString);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1240 goto Lreturn;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1241 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1242 if (!isascii(c))
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1243 {
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1244 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1245 if (isUnicodeNewlineChar(c))
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1246 goto case '\n';
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1247 encodeUTF8(buffer, c);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1248 continue;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1249 }
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1250 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1251 assert(isascii(c));
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1252 buffer ~= c;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1253 }
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
1254 assert(0);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1255 }
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1256
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1257 void scanCharacterLiteral(ref Token t)
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1258 {
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1259 assert(*p == '\'');
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1260 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1261 t.type = TOK.CharLiteral;
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1262 switch (*p)
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1263 {
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1264 case '\\':
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1265 t.dchar_ = scanEscapeSequence();
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1266 break;
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1267 case '\'':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1268 error(t.start, MID.EmptyCharacterLiteral);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1269 break;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1270 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1271 if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1272 break;
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1273 uint c = *p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1274 if (!isascii(c))
49
7f0fa15dcffc - Renamed function.
aziz
parents: 48
diff changeset
1275 c = decodeUTF8();
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1276 t.dchar_ = c;
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1277 ++p;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1278 }
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1279
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1280 if (*p == '\'')
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1281 ++p;
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1282 else
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1283 error(t.start, MID.UnterminatedCharacterLiteral);
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1284 t.end = p;
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1285 }
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1286
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1287 void scanRawStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1288 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1289 assert(*p == '`' || *p == '"' && p[-1] == 'r');
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1290 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1291 auto tokenLineBegin = lineBegin;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1292 t.type = TOK.String;
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1293 uint delim = *p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1294 char[] buffer;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1295 uint c;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1296 while (1)
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1297 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1298 c = *++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1299 switch (c)
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1300 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1301 case '\r':
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1302 if (p[1] == '\n')
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1303 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1304 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1305 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1306 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1307 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1308 setLineBegin(p+1);
52
f65a83c27638 - Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well.
aziz
parents: 51
diff changeset
1309 break;
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1310 case '`':
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1311 case '"':
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1312 if (c == delim)
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1313 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1314 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1315 t.pf = scanPostfix();
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1316 Lreturn:
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1317 t.str = buffer ~ '\0';
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1318 t.end = p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1319 return;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1320 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1321 break;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1322 case 0, _Z_:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1323 error(tokenLineNum, tokenLineBegin, t.start,
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1324 delim == 'r' ? MID.UnterminatedRawString : MID.UnterminatedBackQuoteString);
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1325 goto Lreturn;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1326 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1327 if (!isascii(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1328 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1329 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1330 if (isUnicodeNewlineChar(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1331 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1332 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1333 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1334 }
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1335 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1336 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1337 buffer ~= c;
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1338 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1339 assert(0);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1340 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1341
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1342 void scanHexStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1343 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1344 assert(p[0] == 'x' && p[1] == '"');
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1345 t.type = TOK.String;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1346
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1347 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1348 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1349
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1350 uint c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1351 ubyte[] buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1352 ubyte h; // hex number
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1353 uint n; // number of hex digits
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1354
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1355 ++p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1356 assert(*p == '"');
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1357 while (1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1358 {
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1359 c = *++p;
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1360 switch (c)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1361 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1362 case '"':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1363 if (n & 1)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1364 error(tokenLineNum, tokenLineBegin, t.start, MID.OddNumberOfDigitsInHexString);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1365 ++p;
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1366 t.pf = scanPostfix();
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1367 Lreturn:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1368 t.str = cast(string) (buffer ~= 0);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1369 t.end = p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1370 return;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1371 case '\r':
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1372 if (p[1] == '\n')
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1373 ++p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1374 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1375 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1376 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1377 setLineBegin(p+1);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1378 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1379 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1380 if (ishexad(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1381 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1382 if (c <= '9')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1383 c -= '0';
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1384 else if (c <= 'F')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1385 c -= 'A' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1386 else
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1387 c -= 'a' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1388
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1389 if (n & 1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1390 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1391 h <<= 4;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1392 h |= c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1393 buffer ~= h;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1394 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1395 else
112
004d98df65af - Implemented parseInterfaceDeclaration().
aziz
parents: 105
diff changeset
1396 h = cast(ubyte)c;
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1397 ++n;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1398 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1399 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1400 else if (isspace(c))
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
1401 continue; // Skip spaces.
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
1402 else if (isEOF(c))
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1403 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1404 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedHexString);
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1405 t.pf = 0;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1406 goto Lreturn;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1407 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1408 else
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1409 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1410 auto errorAt = p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1411 if (!isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1412 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1413 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1414 if (isUnicodeNewlineChar(c))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1415 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1416 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1417 error(errorAt, MID.NonHexCharInHexString, cast(dchar)c);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1418 }
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1419 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1420 }
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1421 assert(0);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1422 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1423
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1424 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1425 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1426 void scanDelimitedStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1427 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1428 assert(p[0] == 'q' && p[1] == '"');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1429 t.type = TOK.String;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1430
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1431 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1432 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1433
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1434 char[] buffer;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1435 dchar opening_delim = 0, // 0 if no nested delimiter or '[', '(', '<', '{'
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1436 closing_delim; // Will be ']', ')', '>', '},
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1437 // the first character of an identifier or
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1438 // any other Unicode/ASCII character.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1439 char[] str_delim; // Identifier delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1440 uint level = 1; // Counter for nestable delimiters.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1441
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1442 ++p; ++p; // Skip q"
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1443 uint c = *p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1444 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1445 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1446 case '(':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1447 opening_delim = c;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1448 closing_delim = ')'; // c + 1
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1449 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1450 case '[', '<', '{':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1451 opening_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1452 closing_delim = c + 2; // Get to closing counterpart. Feature of ASCII table.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1453 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1454 default:
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1455 dchar scanNewline()
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1456 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1457 switch (*p)
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1458 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1459 case '\r':
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1460 if (p[1] == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1461 ++p;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1462 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1463 assert(isNewlineEnd(p));
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1464 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1465 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1466 setLineBegin(p);
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1467 return '\n';
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1468 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1469 if (isUnicodeNewline(p))
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1470 {
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
1471 ++p; ++p;
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
1472 goto case '\n';
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1473 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1474 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1475 return 0;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1476 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1477 // Skip leading newlines:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1478 while (scanNewline() != 0)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1479 {}
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1480 assert(!isNewline(p));
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1481
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1482 char* begin = p;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1483 c = *p;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1484 closing_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1485 // TODO: Check for non-printable characters?
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1486 if (!isascii(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1487 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1488 closing_delim = decodeUTF8();
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1489 if (!isUniAlpha(closing_delim))
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1490 break; // Not an identifier.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1491 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1492 else if (!isidbeg(c))
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1493 break; // Not an identifier.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1494
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1495 // Parse Identifier + EndOfLine
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1496 do
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1497 { c = *++p; }
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
1498 while (isident(c) || !isascii(c) && isUnicodeAlpha())
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1499 // Store identifier
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1500 str_delim = begin[0..p-begin];
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1501 // Scan newline
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1502 if (scanNewline() == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1503 --p; // Go back one because of "c = *++p;" in main loop.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1504 else
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1505 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1506 // TODO: error(p, MID.ExpectedNewlineAfterIdentDelim);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1507 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1508 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1509
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1510 bool checkStringDelim(char* p)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1511 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1512 assert(str_delim.length != 0);
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1513 if (buffer[$-1] == '\n' && // Last character copied to buffer must be '\n'.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1514 end-p >= str_delim.length && // Check remaining length.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1515 p[0..str_delim.length] == str_delim) // Compare.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1516 return true;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1517 return false;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1518 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1519
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1520 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1521 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1522 c = *++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1523 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1524 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1525 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1526 if (p[1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1527 ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1528 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1529 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1530 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1531 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1532 setLineBegin(p+1);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1533 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1534 case 0, _Z_:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1535 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedDelimitedString);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1536 goto Lreturn3;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1537 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1538 if (!isascii(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1539 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1540 auto begin = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1541 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1542 if (isUnicodeNewlineChar(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1543 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1544 if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1545 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1546 if (str_delim.length)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1547 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1548 if (checkStringDelim(begin))
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1549 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1550 p = begin + str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1551 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1552 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1553 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1554 else
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1555 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1556 assert(level == 1);
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1557 --level;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1558 goto Lreturn;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1559 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1560 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1561 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1562 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1563 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1564 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1565 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1566 if (c == opening_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1567 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1568 else if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1569 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1570 if (str_delim.length)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1571 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1572 if (checkStringDelim(p))
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1573 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1574 p += str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1575 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1576 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1577 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1578 else if (--level == 0)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1579 goto Lreturn;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1580 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1581 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1582 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1583 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1584 buffer ~= c;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1585 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1586 Lreturn: // Character delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1587 assert(c == closing_delim);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1588 assert(level == 0);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1589 ++p; // Skip closing delimiter.
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1590 Lreturn2: // String delimiter.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1591 if (*p == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1592 ++p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1593 else
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1594 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1595 // TODO: error(p, MID.ExpectedDblQuoteAfterDelim, str_delim.length ? str_delim : closing_delim~"");
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1596 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1597
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1598 t.pf = scanPostfix();
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1599 Lreturn3: // Error.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1600 t.str = buffer ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1601 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1602 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1603
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1604 void scanTokenStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1605 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1606 assert(p[0] == 'q' && p[1] == '{');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1607 t.type = TOK.String;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1608
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1609 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1610 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1611
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1612 // A guard against changes to particular members:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1613 // this.lineNum_hline and this.errorPath
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1614 ++inTokenString;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1615
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1616 uint lineNum = this.lineNum;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1617 uint level = 1;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1618
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1619 ++p; ++p; // Skip q{
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1620
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1621 auto prev_t = &t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1622 Token* token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1623 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1624 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1625 token = new Token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1626 scan(*token);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1627 // Save the tokens in a doubly linked list.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1628 // Could be useful for various tools.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1629 token.prev = prev_t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1630 prev_t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1631 prev_t = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1632 switch (token.type)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1633 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1634 case TOK.LBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1635 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1636 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1637 case TOK.RBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1638 if (--level == 0)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1639 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1640 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1641 t.next = null;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1642 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1643 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1644 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1645 case TOK.EOF:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1646 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedTokenString);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1647 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1648 t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1649 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1650 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1651 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1652 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1653 break; // Exit loop.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1654 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1655
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1656 assert(token.type == TOK.RBrace || token.type == TOK.EOF);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1657 assert(token.type == TOK.RBrace && t.next is null ||
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1658 token.type == TOK.EOF && t.next !is null);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1659
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1660 char[] buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1661 // token points to } or EOF
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1662 if (token.type == TOK.EOF)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1663 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1664 t.end = token.start;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1665 buffer = t.srcText[2..$].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1666 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1667 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1668 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1669 // Assign to buffer before scanPostfix().
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1670 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1671 buffer = t.srcText[2..$-1].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1672 t.pf = scanPostfix();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1673 t.end = p; // Assign again because of postfix.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1674 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1675 // Convert newlines to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1676 if (lineNum != this.lineNum)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1677 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1678 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1679 uint i, j;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1680 for (; i < buffer.length; ++i)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1681 switch (buffer[i])
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1682 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1683 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1684 if (buffer[i+1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1685 ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1686 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1687 assert(isNewlineEnd(buffer.ptr + i));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1688 buffer[j++] = '\n'; // Convert Newline to '\n'.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1689 break;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1690 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1691 if (isUnicodeNewline(buffer.ptr + i))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1692 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1693 ++i; ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1694 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1695 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1696 buffer[j++] = buffer[i]; // Copy.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1697 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1698 buffer.length = j; // Adjust length.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1699 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1700 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1701 t.str = buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1702
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1703 --inTokenString;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1704 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1705 } // version(D2)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1706
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1707 dchar scanEscapeSequence()
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1708 out(result)
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
1709 { assert(isValidChar(result)); }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1710 body
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1711 {
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1712 assert(*p == '\\');
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1713
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1714 auto sequenceStart = p; // Used for error reporting.
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1715
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1716 ++p;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1717 uint c = char2ev(*p);
82
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1718 if (c)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1719 {
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1720 ++p;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1721 return c;
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1722 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1723
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1724 uint digits = 2;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1725
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1726 switch (*p)
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1727 {
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1728 case 'x':
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1729 assert(c == 0);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1730 while (1)
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1731 {
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1732 ++p;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1733 if (ishexad(*p))
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1734 {
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1735 c *= 16;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1736 if (*p <= '9')
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1737 c += *p - '0';
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1738 else if (*p <= 'F')
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1739 c += *p - 'A' + 10;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1740 else
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1741 c += *p - 'a' + 10;
82
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1742
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1743 if (!--digits)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1744 {
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1745 ++p;
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
1746 if (isValidChar(c))
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1747 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1748
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1749 error(sequenceStart, MID.InvalidUnicodeEscapeSequence, sequenceStart[0..p-sequenceStart]);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1750 break;
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1751 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1752 continue;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1753 }
432
3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 430
diff changeset
1754
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1755 error(sequenceStart, MID.InsufficientHexDigits);
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1756 break;
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1757 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1758 break;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1759 case 'u':
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1760 digits = 4;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1761 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1762 case 'U':
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1763 digits = 8;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1764 goto case 'x';
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1765 default:
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1766 if (isoctal(*p))
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1767 {
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1768 assert(c == 0);
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1769 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1770 ++p;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1771 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1772 return c;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1773 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1774 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1775 ++p;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1776 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1777 return c;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1778 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1779 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1780 ++p;
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1781 return c; // Return valid escape value.
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1782 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1783 else if(*p == '&')
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1784 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1785 if (isalpha(*++p))
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1786 {
272
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1787 auto begin = p;
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1788 while (isalnum(*++p))
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1789 {}
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1790
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1791 if (*p == ';')
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1792 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1793 // Pass entity excluding '&' and ';'.
272
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1794 c = entity2Unicode(begin[0..p - begin]);
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1795 ++p; // Skip ;
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1796 if (c != 0xFFFF)
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1797 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1798 else
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1799 error(sequenceStart, MID.UndefinedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
272
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1800 }
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1801 else
432
3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 430
diff changeset
1802 error(sequenceStart, MID.UnterminatedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1803 }
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1804 else
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1805 error(sequenceStart, MID.InvalidBeginHTMLEntity);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1806 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1807 else if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1808 error(sequenceStart, MID.UndefinedEscapeSequence,
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
1809 isEOF(*p) ? `\EOF` : `\NewLine`);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1810 else
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1811 {
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1812 char[] str = `\`;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1813 if (isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1814 str ~= *p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1815 else
387
ad0cbd1c8881 Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 386
diff changeset
1816 encodeUTF8(str, decodeUTF8());
ad0cbd1c8881 Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 386
diff changeset
1817 ++p;
ad0cbd1c8881 Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 386
diff changeset
1818 // TODO: check for unprintable character?
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1819 error(sequenceStart, MID.UndefinedEscapeSequence, str);
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1820 }
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1821 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1822 return REPLACEMENT_CHAR; // Error: return replacement character.
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1823 }
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1824
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1825 /*
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1826 IntegerLiteral:= (Dec|Hex|Bin|Oct)Suffix?
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1827 Dec:= (0|[1-9][0-9_]*)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1828 Hex:= 0[xX] HexDigits
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1829 Bin:= 0[bB][01_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1830 Oct:= 0[0-7_]+
68
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
1831 Suffix:= (L[uU]?|[uU]L?)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1832 HexDigits:= [0-9a-zA-Z_]+
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1833
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1834 Invalid: "0b_", "0x_", "._"
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1835 */
15
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
1836 void scanNumber(ref Token t)
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
1837 {
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1838 ulong ulong_;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1839 bool overflow;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1840 bool isDecimal;
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1841 size_t digits;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1842
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1843 if (*p != '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1844 goto LscanInteger;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1845 ++p; // skip zero
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1846 // check for xX bB ...
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1847 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1848 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1849 case 'x','X':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1850 goto LscanHex;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1851 case 'b','B':
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1852 goto LscanBinary;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1853 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1854 if (p[1] == 'i')
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1855 goto LscanReal; // 0Li
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1856 break; // 0L
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1857 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1858 if (p[1] == '.')
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1859 break; // 0..
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1860 // 0.
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1861 case 'i','f','F', // Imaginary and float literal suffixes.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1862 'e', 'E': // Float exponent.
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1863 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1864 default:
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1865 if (*p == '_')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1866 goto LscanOctal; // 0_
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1867 else if (isdigit(*p))
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1868 {
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1869 if (*p == '8' || *p == '9')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1870 goto Loctal_hasDecimalDigits; // 08 or 09
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1871 else
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1872 goto Loctal_enter_loop; // 0[0-7]
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1873 }
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1874 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1875
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1876 // Number 0
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1877 assert(p[-1] == '0');
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1878 assert(*p != '_' && !isdigit(*p));
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1879 assert(ulong_ == 0);
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1880 isDecimal = true;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1881 goto Lfinalize;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1882
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1883 LscanInteger:
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1884 assert(*p != 0 && isdigit(*p));
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1885 isDecimal = true;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1886 goto Lenter_loop_int;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1887 while (1)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1888 {
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1889 if (*++p == '_')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1890 continue;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1891 if (!isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1892 break;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1893 Lenter_loop_int:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1894 if (ulong_ < ulong.max/10 || (ulong_ == ulong.max/10 && *p <= '5'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1895 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1896 ulong_ *= 10;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1897 ulong_ += *p - '0';
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1898 continue;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1899 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1900 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1901 overflow = true;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1902 while (isdigit(*++p)) {}
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1903 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1904 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1905
61
512cd2248dfc - Fix: issueing error on hexadecimal number overflow.
aziz
parents: 60
diff changeset
1906 // The number could be a float, so check overflow below.
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1907 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1908 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1909 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1910 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1911 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1912 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1913 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1914 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1915 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1916 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1917 goto LscanReal;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1918 default:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1919 }
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1920
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1921 if (overflow)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1922 error(t.start, MID.OverflowDecimalNumber);
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1923
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1924 assert((isdigit(p[-1]) || p[-1] == '_') && !isdigit(*p) && *p != '_');
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1925 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1926
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1927 LscanHex:
61
512cd2248dfc - Fix: issueing error on hexadecimal number overflow.
aziz
parents: 60
diff changeset
1928 assert(digits == 0);
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
1929 assert(*p == 'x' || *p == 'X');
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1930 while (1)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1931 {
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1932 if (*++p == '_')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1933 continue;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1934 if (!ishexad(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1935 break;
61
512cd2248dfc - Fix: issueing error on hexadecimal number overflow.
aziz
parents: 60
diff changeset
1936 ++digits;
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1937 ulong_ *= 16;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1938 if (*p <= '9')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1939 ulong_ += *p - '0';
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1940 else if (*p <= 'F')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1941 ulong_ += *p - 'A' + 10;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1942 else
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1943 ulong_ += *p - 'a' + 10;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1944 }
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1945
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
1946 assert(ishexad(p[-1]) || p[-1] == '_' || p[-1] == 'x' || p[-1] == 'X');
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1947 assert(!ishexad(*p) && *p != '_');
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1948
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1949 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1950 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1951 case '.':
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1952 if (p[1] == '.')
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1953 break;
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1954 case 'p', 'P':
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1955 return scanHexReal(t);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1956 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1957 }
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1958
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1959 if (digits == 0 || digits > 16)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1960 error(t.start, digits == 0 ? MID.NoDigitsInHexNumber : MID.OverflowHexNumber);
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
1961
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1962 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1963
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1964 LscanBinary:
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1965 assert(digits == 0);
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
1966 assert(*p == 'b' || *p == 'B');
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1967 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1968 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1969 if (*++p == '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1970 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1971 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1972 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1973 }
367
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1974 else if (*p == '1')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1975 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1976 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1977 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1978 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1979 }
367
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1980 else if (*p == '_')
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1981 continue;
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1982 else
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1983 break;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1984 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1985
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1986 if (digits == 0 || digits > 64)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1987 error(t.start, digits == 0 ? MID.NoDigitsInBinNumber : MID.OverflowBinaryNumber);
59
3e594725899a - Issuing error when no digits were found in hex and binary numbers.
aziz
parents: 58
diff changeset
1988
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
1989 assert(p[-1] == '0' || p[-1] == '1' || p[-1] == '_' || p[-1] == 'b' || p[-1] == 'B', p[-1] ~ "");
367
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
1990 assert( !(*p == '0' || *p == '1' || *p == '_') );
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1991 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1992
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1993 LscanOctal:
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1994 assert(*p == '_');
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1995 while (1)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1996 {
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1997 if (*++p == '_')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1998 continue;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
1999 if (!isoctal(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2000 break;
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2001 Loctal_enter_loop:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2002 if (ulong_ < ulong.max/2 || (ulong_ == ulong.max/2 && *p <= '1'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2003 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2004 ulong_ *= 8;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2005 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2006 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2007 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2008 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2009 overflow = true;
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2010 while (isoctal(*++p)) {}
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2011 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2012 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2013
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2014 bool hasDecimalDigits;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2015 if (isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2016 {
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2017 Loctal_hasDecimalDigits:
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2018 hasDecimalDigits = true;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2019 while (isdigit(*++p)) {}
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2020 }
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2021
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2022 // The number could be a float, so check errors below.
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2023 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2024 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2025 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2026 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2027 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2028 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2029 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2030 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2031 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2032 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2033 goto LscanReal;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2034 default:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2035 }
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2036
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2037 if (hasDecimalDigits)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2038 error(t.start, MID.OctalNumberHasDecimals);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2039
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2040 if (overflow)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2041 error(t.start, MID.OverflowOctalNumber);
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2042 // goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2043
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2044 Lfinalize:
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2045 enum Suffix
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2046 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2047 None = 0,
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2048 Unsigned = 1,
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2049 Long = 2
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2050 }
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2051
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2052 // Scan optional suffix: L, Lu, LU, u, uL, U or UL.
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2053 Suffix suffix;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2054 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2055 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2056 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2057 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2058 case 'L':
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2059 if (suffix & Suffix.Long)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2060 break;
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2061 suffix |= Suffix.Long;
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2062 ++p;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2063 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2064 case 'u', 'U':
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2065 if (suffix & Suffix.Unsigned)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2066 break;
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2067 suffix |= Suffix.Unsigned;
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2068 ++p;
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2069 continue;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2070 default:
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2071 break;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2072 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2073 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2074 }
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2075
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2076 // Determine type of Integer.
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2077 switch (suffix)
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2078 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2079 case Suffix.None:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2080 if (ulong_ & 0x8000_0000_0000_0000)
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2081 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2082 if (isDecimal)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2083 error(t.start, MID.OverflowDecimalSign);
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2084 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2085 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2086 else if (ulong_ & 0xFFFF_FFFF_0000_0000)
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2087 t.type = TOK.Int64;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2088 else if (ulong_ & 0x8000_0000)
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2089 t.type = isDecimal ? TOK.Int64 : TOK.Uint32;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2090 else
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2091 t.type = TOK.Int32;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2092 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2093 case Suffix.Unsigned:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2094 if (ulong_ & 0xFFFF_FFFF_0000_0000)
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2095 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2096 else
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2097 t.type = TOK.Uint32;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2098 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2099 case Suffix.Long:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2100 if (ulong_ & 0x8000_0000_0000_0000)
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2101 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2102 if (isDecimal)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2103 error(t.start, MID.OverflowDecimalSign);
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2104 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2105 }
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2106 else
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2107 t.type = TOK.Int64;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2108 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2109 case Suffix.Unsigned | Suffix.Long:
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2110 t.type = TOK.Uint64;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2111 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2112 default:
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2113 assert(0);
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2114 }
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2115 t.ulong_ = ulong_;
15
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
2116 t.end = p;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2117 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2118 LscanReal:
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2119 scanReal(t);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2120 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2121 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2122
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2123 /*
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2124 FloatLiteral:= Float[fFL]?i?
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2125 Float:= DecFloat | HexFloat
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2126 DecFloat:= ([0-9][0-9_]*[.][0-9_]*DecExponent?) | [.][0-9][0-9_]*DecExponent? | [0-9][0-9_]*DecExponent
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2127 DecExponent:= [eE][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2128 HexFloat:= 0[xX](HexDigits[.]HexDigits | [.][0-9a-zA-Z]HexDigits? | HexDigits)HexExponent
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2129 HexExponent:= [pP][+-]?[0-9][0-9_]*
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2130 */
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2131 void scanReal(ref Token t)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2132 {
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2133 if (*p == '.')
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2134 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2135 assert(p[1] != '.');
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2136 // This function was called by scan() or scanNumber().
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2137 while (isdigit(*++p) || *p == '_') {}
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2138 }
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2139 else
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2140 // This function was called by scanNumber().
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2141 assert(delegate ()
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2142 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2143 switch (*p)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2144 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2145 case 'L':
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2146 if (p[1] != 'i')
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2147 return false;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2148 case 'i', 'f', 'F', 'e', 'E':
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2149 return true;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2150 default:
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2151 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2152 return false;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2153 }()
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2154 );
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2155
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2156 // Scan exponent.
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2157 if (*p == 'e' || *p == 'E')
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2158 {
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2159 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2160 if (*p == '-' || *p == '+')
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2161 ++p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2162 if (isdigit(*p))
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2163 while (isdigit(*++p) || *p == '_') {}
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2164 else
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2165 error(t.start, MID.FloatExpMustStartWithDigit);
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2166 }
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2167
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2168 // Copy whole number and remove underscores from buffer.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2169 char[] buffer = t.start[0..p-t.start].dup;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2170 uint j;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2171 foreach (c; buffer)
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2172 if (c != '_')
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2173 buffer[j++] = c;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2174 buffer.length = j; // Adjust length.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2175 buffer ~= 0; // Terminate for C functions.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2176
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2177 finalizeFloat(t, buffer);
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2178 }
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2179
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2180 void scanHexReal(ref Token t)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2181 {
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2182 assert(*p == '.' || *p == 'p' || *p == 'P');
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2183 MID mid;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2184 if (*p == '.')
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2185 while (ishexad(*++p) || *p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2186 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2187 // Decimal exponent is required.
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2188 if (*p != 'p' && *p != 'P')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2189 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2190 mid = MID.HexFloatExponentRequired;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2191 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2192 }
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2193 // Scan exponent
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2194 assert(*p == 'p' || *p == 'P');
393
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2195 ++p;
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2196 if (*p == '+' || *p == '-')
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2197 ++p;
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2198 if (!isdigit(*p))
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2199 {
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2200 mid = MID.HexFloatExpMustStartWithDigit;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2201 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2202 }
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2203 while (isdigit(*++p) || *p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2204 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2205 // Copy whole number and remove underscores from buffer.
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2206 char[] buffer = t.start[0..p-t.start].dup;
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2207 uint j;
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2208 foreach (c; buffer)
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2209 if (c != '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2210 buffer[j++] = c;
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2211 buffer.length = j; // Adjust length.
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2212 buffer ~= 0; // Terminate for C functions.
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2213 finalizeFloat(t, buffer);
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2214 return;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2215 Lerr:
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2216 t.type = TOK.Float32;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2217 t.end = p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2218 error(t.start, mid);
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2219 }
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2220
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2221 void finalizeFloat(ref Token t, string buffer)
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2222 {
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2223 assert(buffer[$-1] == 0);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2224 // Float number is well-formed. Check suffixes and do conversion.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2225 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2226 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2227 case 'f', 'F':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2228 t.type = TOK.Float32;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2229 t.float_ = strtof(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2230 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2231 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2232 case 'L':
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2233 t.type = TOK.Float80;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2234 t.real_ = strtold(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2235 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2236 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2237 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2238 t.type = TOK.Float64;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2239 t.double_ = strtod(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2240 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2241 if (*p == 'i')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2242 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2243 ++p;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2244 t.type += 3; // Switch to imaginary counterpart.
538
d0d40bcca9c6 Added assert() to Lexer.finalizeFloat().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 532
diff changeset
2245 assert(t.type == TOK.Imaginary32 ||
d0d40bcca9c6 Added assert() to Lexer.finalizeFloat().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 532
diff changeset
2246 t.type == TOK.Imaginary64 ||
d0d40bcca9c6 Added assert() to Lexer.finalizeFloat().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 532
diff changeset
2247 t.type == TOK.Imaginary80);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2248 }
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
2249 if (errno() == ERANGE)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2250 error(t.start, MID.OverflowFloatNumber);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2251 t.end = p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2252 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2253
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2254 /// Scan special token: #line Integer [Filespec] EndOfLine
360
b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence().
aziz
parents: 350
diff changeset
2255 void scanSpecialTokenSequence(ref Token t)
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2256 {
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
2257 assert(*p == '#');
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2258 t.type = TOK.HashLine;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
2259 t.setWhitespaceFlag();
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2260
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2261 MID mid;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2262 auto errorAtColumn = p;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2263
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2264 ++p;
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2265 if (p[0] != 'l' || p[1] != 'i' || p[2] != 'n' || p[3] != 'e')
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2266 {
370
ae4afb66768f - Renamed findModule() to findModulePath().
aziz
parents: 367
diff changeset
2267 mid = MID.ExpectedIdentifierSTLine;
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2268 goto Lerr;
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2269 }
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2270 p += 3;
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2271
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2272 // TODO: #line58"path/file" is legal. Require spaces?
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2273 // State.Space could be used for that purpose.
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2274 enum State
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2275 { /+Space,+/ Integer, Filespec, End }
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2276
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2277 State state = State.Integer;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2278
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2279 while (!isEndOfLine(++p))
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2280 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2281 if (isspace(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2282 continue;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2283 if (state == State.Integer)
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2284 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2285 if (!isdigit(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2286 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2287 errorAtColumn = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2288 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2289 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2290 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2291 t.tokLineNum = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2292 scan(*t.tokLineNum);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2293 if (t.tokLineNum.type != TOK.Int32 && t.tokLineNum.type != TOK.Uint32)
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2294 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2295 errorAtColumn = t.tokLineNum.start;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2296 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2297 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2298 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2299 --p; // Go one back because scan() advanced p past the integer.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2300 state = State.Filespec;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2301 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2302 else if (state == State.Filespec)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2303 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2304 if (*p != '"')
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2305 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2306 errorAtColumn = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2307 mid = MID.ExpectedFilespec;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2308 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2309 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2310 t.tokLineFilespec = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2311 t.tokLineFilespec.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2312 t.tokLineFilespec.type = TOK.Filespec;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
2313 t.tokLineFilespec.setWhitespaceFlag();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2314 while (*++p != '"')
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2315 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2316 if (isEndOfLine(p))
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2317 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2318 errorAtColumn = t.tokLineFilespec.start;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2319 mid = MID.UnterminatedFilespec;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2320 t.tokLineFilespec.end = p;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2321 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2322 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2323 isascii(*p) || decodeUTF8();
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2324 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2325 auto start = t.tokLineFilespec.start +1; // +1 skips '"'
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2326 t.tokLineFilespec.str = start[0 .. p - start];
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2327 t.tokLineFilespec.end = p + 1;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2328 state = State.End;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2329 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2330 else/+ if (state == State.End)+/
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2331 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2332 mid = MID.UnterminatedSpecialToken;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2333 goto Lerr;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2334 }
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2335 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2336 assert(isEndOfLine(p));
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2337
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2338 if (state == State.Integer)
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2339 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2340 errorAtColumn = p;
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2341 mid = MID.ExpectedIntegerAfterSTLine;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2342 goto Lerr;
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2343 }
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2344
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2345 // Evaluate #line only when not in token string.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2346 if (!inTokenString && t.tokLineNum)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2347 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2348 this.lineNum_hline = this.lineNum - t.tokLineNum.uint_ + 1;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2349 if (t.tokLineFilespec)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2350 this.errorPath = t.tokLineFilespec.str;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2351 }
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2352 t.end = p;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2353
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2354 return;
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2355 Lerr:
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2356 t.end = p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2357 error(errorAtColumn, mid);
371
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer.
aziz
parents: 370
diff changeset
2358 }
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer.
aziz
parents: 370
diff changeset
2359
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2360 /++
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2361 Insert an empty dummy token before t.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2362 Useful in the parsing phase for representing a node in the AST
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2363 that doesn't consume an actual token from the source text.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2364 +/
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2365 Token* insertEmptyTokenBefore(Token* t)
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2366 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2367 assert(t !is null && t.prev !is null);
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2368 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type));
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2369 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type));
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2370
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2371 auto prev_t = t.prev;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2372 auto new_t = new Token;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2373 new_t.type = TOK.Empty;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2374 new_t.start = new_t.end = prev_t.end;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2375 // Link in new token.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2376 prev_t.next = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2377 new_t.prev = prev_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2378 new_t.next = t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2379 t.prev = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2380 return new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2381 }
105
df34ec47fb81 - Added getState() method and State struct to Lexer.
aziz
parents: 103
diff changeset
2382
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2383 uint errorLineNumber(uint lineNum)
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
2384 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2385 return lineNum - this.lineNum_hline;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
2386 }
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
2387
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2388 void error(char* columnPos, MID mid, ...)
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2389 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2390 error_(this.lineNum, this.lineBegin, columnPos, mid, _arguments, _argptr);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2391 }
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2392
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2393 void error(uint lineNum, char* lineBegin, char* columnPos, MID mid, ...)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2394 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2395 error_(lineNum, lineBegin, columnPos, mid, _arguments, _argptr);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2396 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2397
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2398 void error_(uint lineNum, char* lineBegin, char* columnPos, MID mid,
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2399 TypeInfo[] _arguments, void* _argptr)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2400 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2401 lineNum = this.errorLineNumber(lineNum);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2402 auto location = new Location(errorPath, lineNum, lineBegin, columnPos);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2403 auto msg = Format(_arguments, _argptr, GetMsg(mid));
520
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2404 auto error = new LexerError(location, msg);
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2405 errors ~= error;
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2406 if (infoMan !is null)
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2407 infoMan ~= error;
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2408 }
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2409
306
9c866aadcb5b - Moved code out of main() to separate functions.
aziz
parents: 272
diff changeset
2410 Token* getTokens()
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
2411 {
306
9c866aadcb5b - Moved code out of main() to separate functions.
aziz
parents: 272
diff changeset
2412 while (nextToken() != TOK.EOF)
9c866aadcb5b - Moved code out of main() to separate functions.
aziz
parents: 272
diff changeset
2413 {}
9c866aadcb5b - Moved code out of main() to separate functions.
aziz
parents: 272
diff changeset
2414 return head;
3
4bbce78bfb1e - Added TOK enum.
aziz
parents: 2
diff changeset
2415 }
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2416
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2417 /// Scan the whole text until EOF is encountered.
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2418 void scanAll()
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2419 {
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2420 while (nextToken() != TOK.EOF)
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2421 {}
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2422 }
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2423
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2424 /// HEAD -> Newline -> First Token
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2425 Token* firstToken()
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2426 {
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2427 return this.head.next.next;
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2428 }
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2429
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2430 static void loadKeywords(ref Identifier[string] table)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2431 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2432 foreach(k; keywords)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2433 table[k.str] = k;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2434 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2435
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2436 /// Returns true if str is a valid D identifier.
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2437 static bool isIdentifierString(char[] str)
365
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2438 {
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2439 if (str.length == 0 || isdigit(str[0]))
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2440 return false;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2441 size_t idx;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2442 do
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2443 {
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2444 auto c = dil.Unicode.decode(str, idx);
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2445 if (c == ERROR_CHAR || !(isident(c) || !isascii(c) && isUniAlpha(c)))
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2446 return false;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2447 } while (idx < str.length)
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2448 return true;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2449 }
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2450
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2451 /// Returns true if str is a keyword or a special token (__FILE__, __LINE__ etc.)
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2452 static bool isReservedIdentifier(char[] str)
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2453 {
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2454 if (str.length == 0)
382
66477017cb95 Applied fixes to isNonReservedIdentifier().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 371
diff changeset
2455 return false;
66477017cb95 Applied fixes to isNonReservedIdentifier().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 371
diff changeset
2456
365
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2457 static Identifier[string] reserved_ids_table;
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2458 if (reserved_ids_table is null)
499
52447db67938 Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 498
diff changeset
2459 Lexer.loadKeywords(reserved_ids_table);
365
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2460
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2461 if (!isIdentifierString(str))
365
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2462 return false;
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2463
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2464 return (str in reserved_ids_table) !is null;
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2465 }
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2466
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2467 /++
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2468 Returns true if the current character to be decoded is
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2469 a Unicode alpha character.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2470 The current pointer 'p' is not advanced if false is returned.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2471 +/
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2472 bool isUnicodeAlpha()
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2473 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2474 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2475 char* p = this.p;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2476 dchar d = *p;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2477 ++p; // Move to second byte.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2478 // Error if second byte is not a trail byte.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2479 if (!isTrailByte(*p))
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2480 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2481 // Check for overlong sequences.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2482 switch (d)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2483 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2484 case 0xE0, 0xF0, 0xF8, 0xFC:
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2485 if ((*p & d) == 0x80)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2486 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2487 default:
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2488 if ((d & 0xFE) == 0xC0) // 1100000x
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2489 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2490 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2491 const char[] checkNextByte = "if (!isTrailByte(*++p))"
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2492 " return false;";
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2493 const char[] appendSixBits = "d = (d << 6) | *p & 0b0011_1111;";
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2494 // Decode
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2495 if ((d & 0b1110_0000) == 0b1100_0000)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2496 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2497 d &= 0b0001_1111;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2498 mixin(appendSixBits);
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2499 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2500 else if ((d & 0b1111_0000) == 0b1110_0000)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2501 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2502 d &= 0b0000_1111;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2503 mixin(appendSixBits ~
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2504 checkNextByte ~ appendSixBits);
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2505 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2506 else if ((d & 0b1111_1000) == 0b1111_0000)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2507 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2508 d &= 0b0000_0111;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2509 mixin(appendSixBits ~
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2510 checkNextByte ~ appendSixBits ~
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2511 checkNextByte ~ appendSixBits);
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2512 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2513 else
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2514 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2515
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2516 assert(isTrailByte(*p));
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2517 if (!isValidChar(d) || !isUniAlpha(d))
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2518 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2519 // Only advance pointer if this is a Unicode alpha character.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2520 this.p = p;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2521 return true;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2522 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2523
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2524 /// Decodes the next UTF-8 sequence.
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2525 dchar decodeUTF8()
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2526 {
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2527 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2528 char* p = this.p;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2529 dchar d = *p;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2530
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2531 ++p; // Move to second byte.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2532 // Error if second byte is not a trail byte.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2533 if (!isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2534 goto Lerr2;
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2535
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2536 // Check for overlong sequences.
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2537 switch (d)
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2538 {
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2539 case 0xE0, // 11100000 100xxxxx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2540 0xF0, // 11110000 1000xxxx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2541 0xF8, // 11111000 10000xxx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2542 0xFC: // 11111100 100000xx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2543 if ((*p & d) == 0x80)
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2544 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2545 default:
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2546 if ((d & 0xFE) == 0xC0) // 1100000x
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2547 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2548 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2549
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2550 const char[] checkNextByte = "if (!isTrailByte(*++p))"
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2551 " goto Lerr2;";
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2552 const char[] appendSixBits = "d = (d << 6) | *p & 0b0011_1111;";
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2553
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2554 // Decode
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2555 if ((d & 0b1110_0000) == 0b1100_0000)
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2556 { // 110xxxxx 10xxxxxx
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2557 d &= 0b0001_1111;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2558 mixin(appendSixBits);
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2559 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2560 else if ((d & 0b1111_0000) == 0b1110_0000)
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2561 { // 1110xxxx 10xxxxxx 10xxxxxx
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2562 d &= 0b0000_1111;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2563 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2564 checkNextByte ~ appendSixBits);
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2565 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2566 else if ((d & 0b1111_1000) == 0b1111_0000)
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2567 { // 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2568 d &= 0b0000_0111;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2569 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2570 checkNextByte ~ appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2571 checkNextByte ~ appendSixBits);
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2572 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2573 else
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2574 // 5 and 6 byte UTF-8 sequences are not allowed yet.
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2575 // 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2576 // 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2577 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2578
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2579 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2580
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2581 if (!isValidChar(d))
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2582 {
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2583 Lerr:
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2584 // Three cases:
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2585 // *) the UTF-8 sequence was successfully decoded but the resulting
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2586 // character is invalid.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2587 // p points to last trail byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2588 // *) the UTF-8 sequence is overlong.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2589 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2590 // *) the UTF-8 sequence has more than 4 bytes or starts with
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2591 // a trail byte.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2592 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2593 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2594 // Move to next ASCII character or lead byte of a UTF-8 sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2595 while (p < (end-1) && isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2596 ++p;
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2597 --p;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2598 assert(!isTrailByte(p[1]));
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2599 Lerr2:
430
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 426
diff changeset
2600 d = REPLACEMENT_CHAR;
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2601 error(this.p, MID.InvalidUTF8Sequence);
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2602 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2603
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2604 this.p = p;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2605 return d;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2606 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2607
519
9ebc799c7dc5 Fixes in dil.Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 514
diff changeset
2608 static void encodeUTF8(ref char[] str, dchar d)
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2609 {
430
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 426
diff changeset
2610 assert(!isascii(d), "check for ASCII char before calling encodeUTF8().");
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2611 assert(isValidChar(d), "check if character is valid before calling encodeUTF8().");
430
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 426
diff changeset
2612
519
9ebc799c7dc5 Fixes in dil.Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 514
diff changeset
2613 char[6] b = void;
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2614 if (d < 0x800)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2615 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2616 b[0] = 0xC0 | (d >> 6);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2617 b[1] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2618 str ~= b[0..2];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2619 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2620 else if (d < 0x10000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2621 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2622 b[0] = 0xE0 | (d >> 12);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2623 b[1] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2624 b[2] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2625 str ~= b[0..3];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2626 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2627 else if (d < 0x200000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2628 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2629 b[0] = 0xF0 | (d >> 18);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2630 b[1] = 0x80 | ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2631 b[2] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2632 b[3] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2633 str ~= b[0..4];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2634 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2635 /+ // There are no 5 and 6 byte UTF-8 sequences yet.
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2636 else if (d < 0x4000000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2637 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2638 b[0] = 0xF8 | (d >> 24);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2639 b[1] = 0x80 | ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2640 b[2] = 0x80 | ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2641 b[3] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2642 b[4] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2643 str ~= b[0..5];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2644 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2645 else if (d < 0x80000000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2646 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2647 b[0] = 0xFC | (d >> 30);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2648 b[1] = 0x80 | ((d >> 24) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2649 b[2] = 0x80 | ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2650 b[3] = 0x80 | ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2651 b[4] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2652 b[5] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2653 str ~= b[0..6];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2654 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2655 +/
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2656 else
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2657 assert(0);
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2658 }
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
2659 }
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2660
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2661 unittest
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2662 {
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
2663 Stdout("Testing Lexer.\n");
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2664 struct Pair
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2665 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2666 char[] tokenText;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2667 TOK type;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2668 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2669 static Pair[] pairs = [
494
9a7ca8c56e59 Refactored a few things in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
2670 {"#!äöüß", TOK.Shebang}, {"\n", TOK.Newline},
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2671 {"//çay", TOK.Comment}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2672 {"&", TOK.AndBinary},
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2673 {"/*çağ*/", TOK.Comment}, {"&&", TOK.AndLogical},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2674 {"/+çak+/", TOK.Comment}, {"&=", TOK.AndAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2675 {">", TOK.Greater}, {"+", TOK.Plus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2676 {">=", TOK.GreaterEqual}, {"++", TOK.PlusPlus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2677 {">>", TOK.RShift}, {"+=", TOK.PlusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2678 {">>=", TOK.RShiftAssign}, {"-", TOK.Minus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2679 {">>>", TOK.URShift}, {"--", TOK.MinusMinus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2680 {">>>=", TOK.URShiftAssign}, {"-=", TOK.MinusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2681 {"<", TOK.Less}, {"=", TOK.Assign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2682 {"<=", TOK.LessEqual}, {"==", TOK.Equal},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2683 {"<>", TOK.LorG}, {"~", TOK.Tilde},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2684 {"<>=", TOK.LorEorG}, {"~=", TOK.CatAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2685 {"<<", TOK.LShift}, {"*", TOK.Mul},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2686 {"<<=", TOK.LShiftAssign}, {"*=", TOK.MulAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2687 {"!", TOK.Not}, {"/", TOK.Div},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2688 {"!=", TOK.NotEqual}, {"/=", TOK.DivAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2689 {"!<", TOK.UorGorE}, {"^", TOK.Xor},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2690 {"!>", TOK.UorLorE}, {"^=", TOK.XorAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2691 {"!<=", TOK.UorG}, {"%", TOK.Mod},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2692 {"!>=", TOK.UorL}, {"%=", TOK.ModAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2693 {"!<>", TOK.UorE}, {"(", TOK.LParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2694 {"!<>=", TOK.Unordered}, {")", TOK.RParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2695 {".", TOK.Dot}, {"[", TOK.LBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2696 {"..", TOK.Slice}, {"]", TOK.RBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2697 {"...", TOK.Ellipses}, {"{", TOK.LBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2698 {"|", TOK.OrBinary}, {"}", TOK.RBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2699 {"||", TOK.OrLogical}, {":", TOK.Colon},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2700 {"|=", TOK.OrAssign}, {";", TOK.Semicolon},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2701 {"?", TOK.Question}, {",", TOK.Comma},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2702 {"$", TOK.Dollar}, {"cam", TOK.Identifier},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2703 {"çay", TOK.Identifier}, {".0", TOK.Float64},
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2704 {"0", TOK.Int32}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2705 {"\r", TOK.Newline}, {"\r\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2706 {"\u2028", TOK.Newline}, {"\u2029", TOK.Newline}
40
9d5ceb0f8be9 - Added more tokens for testing.
aziz
parents: 39
diff changeset
2707 ];
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2708
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2709 char[] src;
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2710
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2711 // Join all token texts into a single string.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2712 foreach (i, pair; pairs)
494
9a7ca8c56e59 Refactored a few things in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
2713 if (pair.type == TOK.Comment && pair.tokenText[1] == '/' || // Line comment.
9a7ca8c56e59 Refactored a few things in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
2714 pair.type == TOK.Shebang)
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2715 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2716 assert(pairs[i+1].type == TOK.Newline); // Must be followed by a newline.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2717 src ~= pair.tokenText;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2718 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2719 else
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2720 src ~= pair.tokenText ~ " ";
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2721
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2722 auto lx = new Lexer(src, "");
350
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2723 auto token = lx.getTokens();
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2724
350
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2725 uint i;
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2726 assert(token == lx.head);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2727 assert(token.next.type == TOK.Newline);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2728 token = token.next.next;
350
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2729 do
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2730 {
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2731 assert(i < pairs.length);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2732 assert(token.srcText == pairs[i].tokenText, Format("Scanned '{0}' but expected '{1}'", token.srcText, pairs[i].tokenText));
350
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2733 ++i;
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2734 token = token.next;
4ea6759300cf - Fixed unittests in module Lexer.
aziz
parents: 346
diff changeset
2735 } while (token.type != TOK.EOF)
41
2b7be1d67d4d - Optimized scanner of block comments.
aziz
parents: 40
diff changeset
2736 }
55
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2737
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2738 unittest
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2739 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2740 Stdout("Testing method Lexer.peek()\n");
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2741 string sourceText = "unittest { }";
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2742 auto lx = new Lexer(sourceText, null);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2743
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2744 auto next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2745 lx.peek(next);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2746 assert(next.type == TOK.Newline);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2747 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2748 assert(next.type == TOK.Unittest);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2749 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2750 assert(next.type == TOK.LBrace);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2751 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2752 assert(next.type == TOK.RBrace);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2753 lx.peek(next);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2754 assert(next.type == TOK.EOF);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2755
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2756 lx = new Lexer("", null);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2757 next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2758 lx.peek(next);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2759 assert(next.type == TOK.Newline);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2760 lx.peek(next);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2761 assert(next.type == TOK.EOF);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2762 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2763
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2764 unittest
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2765 {
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2766 // Numbers unittest
68
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2767 // 0L 0ULi 0_L 0_UL 0x0U 0x0p2 0_Fi 0_e2 0_F 0_i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2768 // 0u 0U 0uL 0UL 0L 0LU 0Lu
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2769 // 0Li 0f 0F 0fi 0Fi 0i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2770 // 0b_1_LU 0b1000u
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2771 // 0x232Lu
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2772 }
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2773
55
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2774 /// ASCII character properties table.
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2775 static const int ptable[256] = [
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2776 0, 0, 0, 0, 0, 0, 0, 0, 0,32, 0,32,32, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2777 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2778 32, 0, 0x2200, 0, 0, 0, 0, 0x2700, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2779 7, 7, 7, 7, 7, 7, 7, 7, 6, 6, 0, 0, 0, 0, 0, 0x3f00,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2780 0,12,12,12,12,12,12, 8, 8, 8, 8, 8, 8, 8, 8, 8,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2781 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0, 0x5c00, 0, 0,16,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2782 0, 0x70c, 0x80c,12,12,12, 0xc0c, 8, 8, 8, 8, 8, 8, 8, 0xa08, 8,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2783 8, 8, 0xd08, 8, 0x908, 8, 0xb08, 8, 8, 8, 8, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2784 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2785 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2786 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2787 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2788 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2789 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2790 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2791 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2792 ];
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2793
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2794 enum CProperty
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2795 {
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2796 Octal = 1,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2797 Digit = 1<<1,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2798 Hex = 1<<2,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2799 Alpha = 1<<3,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2800 Underscore = 1<<4,
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2801 Whitespace = 1<<5
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2802 }
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2803
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2804 const uint EVMask = 0xFF00; // Bit mask for escape value
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2805
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2806 private alias CProperty CP;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2807 int isoctal(char c) { return ptable[c] & CP.Octal; }
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2808 int isdigit(char c) { return ptable[c] & CP.Digit; }
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2809 int ishexad(char c) { return ptable[c] & CP.Hex; }
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2810 int isalpha(char c) { return ptable[c] & CP.Alpha; }
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2811 int isalnum(char c) { return ptable[c] & (CP.Alpha | CP.Digit); }
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2812 int isidbeg(char c) { return ptable[c] & (CP.Alpha | CP.Underscore); }
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2813 int isident(char c) { return ptable[c] & (CP.Alpha | CP.Underscore | CP.Digit); }
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2814 int isspace(char c) { return ptable[c] & CP.Whitespace; }
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2815 int char2ev(char c) { return ptable[c] >> 8; /*(ptable[c] & EVMask) >> 8;*/ }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2816 int isascii(uint c) { return c < 128; }
55
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2817
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2818 version(gen_ptable)
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2819 static this()
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2820 {
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2821 alias ptable p;
402
22d65b2bef4f Migrated the code that generates the ptable to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 397
diff changeset
2822 assert(p.length == 256);
55
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2823 // Initialize character properties table.
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2824 for (int i; i < p.length; ++i)
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2825 {
402
22d65b2bef4f Migrated the code that generates the ptable to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 397
diff changeset
2826 p[i] = 0; // Reset
55
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2827 if ('0' <= i && i <= '7')
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2828 p[i] |= CP.Octal;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2829 if ('0' <= i && i <= '9')
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2830 p[i] |= CP.Digit;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2831 if (isdigit(i) || 'a' <= i && i <= 'f' || 'A' <= i && i <= 'F')
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2832 p[i] |= CP.Hex;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2833 if ('a' <= i && i <= 'z' || 'A' <= i && i <= 'Z')
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2834 p[i] |= CP.Alpha;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2835 if (i == '_')
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2836 p[i] |= CP.Underscore;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2837 if (i == ' ' || i == '\t' || i == '\v' || i == '\f')
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2838 p[i] |= CP.Whitespace;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2839 }
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2840 // Store escape sequence values in second byte.
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2841 assert(CProperty.max <= ubyte.max, "character property flags and escape value byte overlap.");
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2842 p['\''] |= 39 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2843 p['"'] |= 34 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2844 p['?'] |= 63 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2845 p['\\'] |= 92 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2846 p['a'] |= 7 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2847 p['b'] |= 8 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2848 p['f'] |= 12 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2849 p['n'] |= 10 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2850 p['r'] |= 13 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2851 p['t'] |= 9 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2852 p['v'] |= 11 << 8;
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2853 // Print a formatted array literal.
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2854 char[] array = "[\n";
402
22d65b2bef4f Migrated the code that generates the ptable to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 397
diff changeset
2855 foreach (i, c; ptable)
55
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2856 {
402
22d65b2bef4f Migrated the code that generates the ptable to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 397
diff changeset
2857 array ~= Format((c>255?" 0x{0:x},":"{0,2},"), c) ~ (((i+1) % 16) ? "":"\n");
55
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2858 }
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2859 array[$-2..$] = "\n]";
402
22d65b2bef4f Migrated the code that generates the ptable to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 397
diff changeset
2860 Stdout(array).newline;
55
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2861 }