Mercurial > projects > dil

annotate src/dil/lexer/Lexer.d @ 829:55c463c57d3a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fixed variable argument parameter issue.
author Aziz K?ksal <aziz.koeksal@gmail.com>
date Tue, 01 Jul 2008 22:52:54 +0200
parents d659f7aa055c
children 451ede0105e0
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
1 /++
8ba2570de175 Initial import.
aziz
parents:
diff changeset
2 Author: Aziz Köksal
249
32d354584b28 - Upgraded license notices to GPL3.
aziz
parents: 239
diff changeset
3 License: GPL3
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
4 +/
576
0df647660e76 Moved Lexer.d to new package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 552
diff changeset
5 module dil.lexer.Lexer;
577
9e811db780a6 Moved LexerFuncs.d to package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 576
diff changeset
6
596
39fac5531b85 Moved dil.Token to dil.lexer.Token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
7 import dil.lexer.Token;
599
c4cdea3a65ba Moved dil.Keywords to dil.lexer.Keywords.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 596
diff changeset
8 import dil.lexer.Keywords;
600
041eae272362 Moved dil.Identifier to dil.lexer.Identifier.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 599
diff changeset
9 import dil.lexer.Identifier;
601
9f96fd51cb41 Moved dil.IdTable to dil.lexer.IdTable.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 600
diff changeset
10 import dil.lexer.IdTable;
327
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
11 import dil.Information;
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
12 import dil.Messages;
a48a987f7515 - Added package dil to import declarations.
aziz
parents: 326
diff changeset
13 import dil.HtmlEntities;
464
325714d8aa6c Added new module with compiler constants to resolve import cycle.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 434
diff changeset
14 import dil.CompilerInfo;
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
15 import dil.Unicode;
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
16 import dil.SourceText;
776
580d4ca9f1ff Added new module dil.Time.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 769
diff changeset
17 import dil.Time;
737
f88b5285b86b Implemented DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 715
diff changeset
18 import common;
f88b5285b86b Implemented DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 715
diff changeset
19
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
20 import tango.stdc.stdlib : strtof, strtod, strtold;
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
21 import tango.stdc.errno : errno, ERANGE;
829
55c463c57d3a Fixed variable argument parameter issue.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 826
diff changeset
22 import tango.core.Vararg;
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
23
577
9e811db780a6 Moved LexerFuncs.d to package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 576
diff changeset
24 public import dil.lexer.Funcs;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
25
776
580d4ca9f1ff Added new module dil.Time.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 769
diff changeset
26 /// The Lexer analyzes the characters of a source text and
580d4ca9f1ff Added new module dil.Time.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 769
diff changeset
27 /// produces a doubly-linked list of tokens.
0
8ba2570de175 Initial import.
aziz
parents:
diff changeset
28 class Lexer
8ba2570de175 Initial import.
aziz
parents:
diff changeset
29 {
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
30 SourceText srcText; /// The source text.
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
31 char* p; /// Points to the current character in the source text.
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
32 char* end; /// Points one character past the end of the source text.
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
33
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
34 Token* head; /// The head of the doubly linked token list.
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
35 Token* tail; /// The tail of the linked list. Set in scan().
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
36 Token* token; /// Points to the current token in the token list.
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
37
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
38 // Members used for error messages:
532
50e64bab9c7a Renamed InformationManager to InfoManager.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 520
diff changeset
39 InfoManager infoMan;
514
6ddff941862a Added new error classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 510
diff changeset
40 LexerError[] errors;
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
41 /// Always points to the first character of the current line.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
42 char* lineBegin;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
43 // Token* newline; /// Current newline token.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
44 uint lineNum = 1; /// Current, actual source text line number.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
45 uint lineNum_hline; /// Line number set by #line.
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
46 uint inTokenString; /// > 0 if inside q{ }
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
47 /// Holds the original file path and the modified one (by #line.)
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
48 NewlineData.FilePaths* filePaths;
103
511a1aa25896 - Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too.
aziz
parents: 94
diff changeset
49
776
580d4ca9f1ff Added new module dil.Time.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 769
diff changeset
50 /// Construct a Lexer object.
580d4ca9f1ff Added new module dil.Time.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 769
diff changeset
51 /// Params:
580d4ca9f1ff Added new module dil.Time.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 769
diff changeset
52 /// srcText = the UTF-8 source code.
580d4ca9f1ff Added new module dil.Time.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 769
diff changeset
53 /// infoMan = used for collecting error messages.
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
54 this(SourceText srcText, InfoManager infoMan = null)
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
55 {
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
56 this.srcText = srcText;
520
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
57 this.infoMan = infoMan;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
58
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
59 assert(text.length && text[$-1] == 0, "source text has no sentinel character");
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
60 this.p = text.ptr;
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
61 this.end = this.p + text.length;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
62 this.lineBegin = this.p;
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
63
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
64 this.head = new Token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
65 this.head.kind = TOK.HEAD;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
66 this.head.start = this.head.end = this.p;
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
67 this.token = this.head;
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
68 // Initialize this.filePaths.
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
69 newFilePath(this.srcText.filePath);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
70 // Add a newline as the first token after the head.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
71 auto newline = new Token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
72 newline.kind = TOK.Newline;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
73 newline.setWhitespaceFlag();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
74 newline.start = newline.end = this.p;
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
75 newline.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
76 newline.newline.oriLineNum = 1;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
77 newline.newline.setLineNum = 0;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
78 // Link in.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
79 this.token.next = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
80 newline.prev = this.token;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
81 this.token = newline;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
82 // this.newline = newline;
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
83 scanShebang();
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
84 }
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
85
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
86 /// The destructor deletes the doubly-linked token list.
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
87 ~this()
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
88 {
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
89 auto token = head.next;
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
90 while (token !is null)
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
91 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
92 assert(token.kind == TOK.EOF ? token == tail && token.next is null : 1);
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
93 delete token.prev;
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
94 token = token.next;
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
95 }
344
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
96 delete tail;
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
97 }
757c86e2c3cc - Added member tail and destructor method to Lexer.
aziz
parents: 343
diff changeset
98
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
99 char[] text()
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
100 {
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
101 return srcText.data;
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
102 }
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
103
776
580d4ca9f1ff Added new module dil.Time.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 769
diff changeset
104 /// The "shebang" may optionally appear once at the beginning of a file.
580d4ca9f1ff Added new module dil.Time.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 769
diff changeset
105 /// Regexp: #![^\EndOfLine]*
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
106 void scanShebang()
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
107 {
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
108 if (*p == '#' && p[1] == '!')
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
109 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
110 auto t = new Token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
111 t.kind = TOK.Shebang;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
112 t.setWhitespaceFlag();
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
113 t.start = p;
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
114 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
115 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
116 isascii(*p) || decodeUTF8();
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
117 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
118 this.token.next = t;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
119 t.prev = this.token;
315
29c33ce6c5bb - Added method scanShebang to class Lexer.
aziz
parents: 309
diff changeset
120 }
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
121 }
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
122
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
123 /// Sets the value of the special token.
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
124 void finalizeSpecialToken(ref Token t)
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
125 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
126 assert(t.srcText[0..2] == "__");
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
127 switch (t.kind)
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
128 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
129 case TOK.FILE:
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
130 t.str = this.filePaths.setPath;
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
131 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
132 case TOK.LINE:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
133 t.uint_ = this.errorLineNumber(this.lineNum);
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
134 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
135 case TOK.DATE,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
136 TOK.TIME,
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
137 TOK.TIMESTAMP:
776
580d4ca9f1ff Added new module dil.Time.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 769
diff changeset
138 auto time_str = Time.toString();
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
139 switch (t.kind)
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
140 {
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
141 case TOK.DATE:
776
580d4ca9f1ff Added new module dil.Time.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 769
diff changeset
142 time_str = Time.month_day(time_str) ~ ' ' ~ Time.year(time_str); break;
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
143 case TOK.TIME:
776
580d4ca9f1ff Added new module dil.Time.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 769
diff changeset
144 time_str = Time.time(time_str); break;
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
145 case TOK.TIMESTAMP:
776
580d4ca9f1ff Added new module dil.Time.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 769
diff changeset
146 break; // time_str is the timestamp.
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
147 default: assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
148 }
776
580d4ca9f1ff Added new module dil.Time.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 769
diff changeset
149 time_str ~= '\0'; // Terminate with a zero.
343
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
150 t.str = time_str;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
151 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
152 case TOK.VENDOR:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
153 t.str = VENDOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
154 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
155 case TOK.VERSION:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
156 t.uint_ = VERSION_MAJOR*1000 + VERSION_MINOR;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
157 break;
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
158 default:
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
159 assert(0);
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
160 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
161 }
95f1b6e43214 - Removed TOK.Special and added an own entry for each special token.
aziz
parents: 327
diff changeset
162
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
163 /// Sets a new file path.
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
164 void newFilePath(char[] newPath)
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
165 {
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
166 auto paths = new NewlineData.FilePaths;
755
90668b83ae5e Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 742
diff changeset
167 paths.oriPath = this.srcText.filePath;
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
168 paths.setPath = newPath;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
169 this.filePaths = paths;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
170 }
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
171
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
172 private void setLineBegin(char* p)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
173 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
174 // Check that we can look behind one character.
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
175 assert((p-1) >= text.ptr && p < end);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
176 // Check that previous character is a newline.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
177 assert(isNewlineEnd(p - 1));
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
178 this.lineBegin = p;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
179 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
180
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
181 /// Scans the next token in the source text.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
182 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
183 /// Creates a new token if t.next is null and appends it to the list.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
184 private void scanNext(ref Token* t)
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
185 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
186 assert(t !is null);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
187 if (t.next)
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
188 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
189 t = t.next;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
190 // if (t.kind == TOK.Newline)
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
191 // this.newline = t;
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
192 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
193 else if (t != this.tail)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
194 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
195 Token* new_t = new Token;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
196 scan(*new_t);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
197 new_t.prev = t;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
198 t.next = new_t;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
199 t = new_t;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
200 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
201 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
202
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
203 /// Advance t one token forward.
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
204 void peek(ref Token* t)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
205 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
206 scanNext(t);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
207 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
208
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
209 /// Advance to the next token in the source text.
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
210 TOK nextToken()
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
211 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
212 scanNext(this.token);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
213 return this.token.kind;
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
214 }
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
215
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
216 /// Returns true if p points to the last character of a Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
217 bool isNewlineEnd(char* p)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
218 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
219 if (*p == '\n' || *p == '\r')
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
220 return true;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
221 if (*p == LS[2] || *p == PS[2])
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
222 if ((p-2) >= text.ptr)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
223 if (p[-1] == LS[1] && p[-2] == LS[0])
434
cac9c8d637ad Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 433
diff changeset
224 return true;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
225 return false;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
226 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
227
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
228 /// The main method which recognizes the characters that make up a token.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
229 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
230 /// Complicated tokens are scanned in separate methods.
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
231 public void scan(ref Token t)
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
232 in
3
4bbce78bfb1e - Added TOK enum.
aziz
parents: 2
diff changeset
233 {
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
234 assert(text.ptr <= p && p < end);
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
235 }
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
236 out
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
237 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
238 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
239 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
240 }
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
241 body
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
242 {
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
243 // Scan whitespace.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
244 if (isspace(*p))
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
245 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
246 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
247 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
248 {}
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
249 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
250
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
251 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
252 uint c = *p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
253 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
254 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
255 // Newline.
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
256 switch (*p)
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
257 {
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
258 case '\r':
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
259 if (p[1] == '\n')
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
260 ++p;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
261 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
262 assert(isNewlineEnd(p));
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
263 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
264 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
265 setLineBegin(p);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
266 // this.newline = &t;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
267 t.kind = TOK.Newline;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
268 t.setWhitespaceFlag();
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
269 t.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
270 t.newline.oriLineNum = lineNum;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
271 t.newline.setLineNum = lineNum_hline;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
272 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
273 return;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
274 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
275 if (isUnicodeNewline(p))
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
276 {
412
fb31af0fda73 Added struct Location, and token2LocTable to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 411
diff changeset
277 ++p; ++p;
fb31af0fda73 Added struct Location, and token2LocTable to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 411
diff changeset
278 goto case '\n';
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
279 }
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
280 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
281 // Identifier or string literal.
13
e5211758b63c - Added isidbeg() function.
aziz
parents: 12
diff changeset
282 if (isidbeg(c))
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
283 {
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
284 if (c == 'r' && p[1] == '"' && ++p)
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
285 return scanRawStringLiteral(t);
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
286 if (c == 'x' && p[1] == '"')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
287 return scanHexStringLiteral(t);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
288 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
289 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
290 if (c == 'q' && p[1] == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
291 return scanDelimitedStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
292 if (c == 'q' && p[1] == '{')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
293 return scanTokenStringLiteral(t);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
294 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
295 // Scan identifier.
12
0989206cf73c - Added code to decode Unicode characters in identifiers.
aziz
parents: 11
diff changeset
296 Lidentifier:
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
297 do
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
298 { c = *++p; }
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
299 while (isident(c) || !isascii(c) && isUnicodeAlpha())
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
300
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
301 t.end = p;
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
302
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
303 auto id = IdTable.lookup(t.srcText);
769
5e3ef1b2011c Added and improved documentation.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 764
diff changeset
304 t.kind = id.kind;
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
305 t.ident = id;
28
3a9daccf7d96 - Added table for identifiers to Lexer.
aziz
parents: 27
diff changeset
306
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
307 if (t.kind == TOK.Identifier || t.isKeyword)
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
308 return;
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
309 else if (t.isSpecialToken)
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
310 finalizeSpecialToken(t);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
311 else if (t.kind == TOK.EOF)
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
312 {
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
313 tail = &t;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
314 assert(t.srcText == "__EOF__");
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
315 }
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
316 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
317 assert(0, "unexpected token type: " ~ Token.toString(t.kind));
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
318 return;
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
319 }
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
320
15
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
321 if (isdigit(c))
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
322 return scanNumber(t);
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
323
8
d4ba94a5a282 - Parsing /* */ comments now.
aziz
parents: 7
diff changeset
324 if (c == '/')
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
325 {
8
d4ba94a5a282 - Parsing /* */ comments now.
aziz
parents: 7
diff changeset
326 c = *++p;
14
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
327 switch(c)
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
328 {
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
329 case '=':
14
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
330 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
331 t.kind = TOK.DivAssign;
14
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
332 t.end = p;
cdf788d8bdaf - Parsing /= now.
aziz
parents: 13
diff changeset
333 return;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
334 case '+':
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
335 return scanNestedComment(t);
32
d7011daa4740 - Added missing commas after the items in the messages table.
aziz
parents: 31
diff changeset
336 case '*':
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
337 return scanBlockComment(t);
32
d7011daa4740 - Added missing commas after the items in the messages table.
aziz
parents: 31
diff changeset
338 case '/':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
339 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
340 isascii(*p) || decodeUTF8();
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
341 t.kind = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
342 t.setWhitespaceFlag();
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
343 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
344 return;
32
d7011daa4740 - Added missing commas after the items in the messages table.
aziz
parents: 31
diff changeset
345 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
346 t.kind = TOK.Div;
10
3ee65d6e39c9 - Parsing // comments now.
aziz
parents: 9
diff changeset
347 t.end = p;
3ee65d6e39c9 - Parsing // comments now.
aziz
parents: 9
diff changeset
348 return;
3ee65d6e39c9 - Parsing // comments now.
aziz
parents: 9
diff changeset
349 }
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
350 }
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
351
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
352 switch (c)
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
353 {
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
354 case '\'':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
355 return scanCharacterLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
356 case '`':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
357 return scanRawStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
358 case '"':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
359 return scanNormalStringLiteral(t);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
360 case '\\':
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
361 char[] buffer;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
362 do
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
363 {
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
364 bool isBinary;
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
365 c = scanEscapeSequence(isBinary);
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
366 if (isascii(c) || isBinary)
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
367 buffer ~= c;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
368 else
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
369 encodeUTF8(buffer, c);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
370 } while (*p == '\\')
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
371 buffer ~= 0;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
372 t.kind = TOK.String;
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
373 t.str = buffer;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
374 t.end = p;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
375 return;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
376 case '>': /* > >= >> >>= >>> >>>= */
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
377 c = *++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
378 switch (c)
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
379 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
380 case '=':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
381 t.kind = TOK.GreaterEqual;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
382 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
383 case '>':
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
384 if (p[1] == '>')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
385 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
386 ++p;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
387 if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
388 { ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
389 t.kind = TOK.URShiftAssign;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
390 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
391 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
392 t.kind = TOK.URShift;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
393 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
394 else if (p[1] == '=')
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
395 {
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
396 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
397 t.kind = TOK.RShiftAssign;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
398 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
399 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
400 t.kind = TOK.RShift;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
401 goto Lcommon;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
402 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
403 t.kind = TOK.Greater;
38
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
404 goto Lcommon2;
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
405 }
640c45aaaaee - Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents: 37
diff changeset
406 assert(0);
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
407 case '<': /* < <= <> <>= << <<= */
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
408 c = *++p;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
409 switch (c)
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
410 {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
411 case '=':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
412 t.kind = TOK.LessEqual;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
413 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
414 case '<':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
415 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
416 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
417 t.kind = TOK.LShiftAssign;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
418 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
419 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
420 t.kind = TOK.LShift;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
421 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
422 case '>':
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
423 if (p[1] == '=') {
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
424 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
425 t.kind = TOK.LorEorG;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
426 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
427 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
428 t.kind = TOK.LorG;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
429 goto Lcommon;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
430 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
431 t.kind = TOK.Less;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
432 goto Lcommon2;
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
433 }
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
434 assert(0);
37
7f3bcb97d017 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 36
diff changeset
435 case '!': /* ! !< !> !<= !>= !<> !<>= */
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
436 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
437 switch (c)
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
438 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
439 case '<':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
440 c = *++p;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
441 if (c == '>')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
442 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
443 if (p[1] == '=') {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
444 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
445 t.kind = TOK.Unordered;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
446 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
447 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
448 t.kind = TOK.UorE;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
449 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
450 else if (c == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
451 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
452 t.kind = TOK.UorG;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
453 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
454 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
455 t.kind = TOK.UorGorE;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
456 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
457 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
458 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
459 case '>':
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
460 if (p[1] == '=')
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
461 {
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
462 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
463 t.kind = TOK.UorL;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
464 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
465 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
466 t.kind = TOK.UorLorE;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
467 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
468 case '=':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
469 t.kind = TOK.NotEqual;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
470 goto Lcommon;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
471 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
472 t.kind = TOK.Not;
35
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
473 goto Lcommon2;
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
474 }
c470b9356e35 - Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents: 34
diff changeset
475 assert(0);
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
476 case '.': /* . .[0-9] .. ... */
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
477 if (p[1] == '.')
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
478 {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
479 ++p;
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
480 if (p[1] == '.') {
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
481 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
482 t.kind = TOK.Ellipses;
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
483 }
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
484 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
485 t.kind = TOK.Slice;
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
486 }
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
487 else if (isdigit(p[1]))
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
488 {
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
489 return scanReal(t);
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
490 }
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
491 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
492 t.kind = TOK.Dot;
22
b05fff8e2ce4 - Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents: 21
diff changeset
493 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
494 case '|': /* | || |= */
23
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
495 c = *++p;
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
496 if (c == '=')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
497 t.kind = TOK.OrAssign;
23
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
498 else if (c == '|')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
499 t.kind = TOK.OrLogical;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
500 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
501 t.kind = TOK.OrBinary;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
502 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
503 }
23
1a7903701a3d - Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents: 22
diff changeset
504 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
505 case '&': /* & && &= */
24
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
506 c = *++p;
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
507 if (c == '=')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
508 t.kind = TOK.AndAssign;
24
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
509 else if (c == '&')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
510 t.kind = TOK.AndLogical;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
511 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
512 t.kind = TOK.AndBinary;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
513 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
514 }
24
903f91163f23 - Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents: 23
diff changeset
515 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
516 case '+': /* + ++ += */
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
517 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
518 if (c == '=')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
519 t.kind = TOK.PlusAssign;
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
520 else if (c == '+')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
521 t.kind = TOK.PlusPlus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
522 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
523 t.kind = TOK.Plus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
524 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
525 }
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
526 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
527 case '-': /* - -- -= */
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
528 c = *++p;
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
529 if (c == '=')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
530 t.kind = TOK.MinusAssign;
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
531 else if (c == '-')
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
532 t.kind = TOK.MinusMinus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
533 else {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
534 t.kind = TOK.Minus;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
535 goto Lcommon2;
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
536 }
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
537 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
538 case '=': /* = == */
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
539 if (p[1] == '=') {
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
540 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
541 t.kind = TOK.Equal;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
542 }
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
543 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
544 t.kind = TOK.Assign;
25
9c866208b3f6 - Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents: 24
diff changeset
545 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
546 case '~': /* ~ ~= */
27
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
547 if (p[1] == '=') {
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
548 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
549 t.kind = TOK.CatAssign;
27
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
550 }
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
551 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
552 t.kind = TOK.Tilde;
27
43b6bf56f0e9 - Added code for parsing CatAssign and Tilde tokens.
aziz
parents: 26
diff changeset
553 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
554 case '*': /* * *= */
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
555 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
556 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
557 t.kind = TOK.MulAssign;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
558 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
559 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
560 t.kind = TOK.Mul;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
561 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
562 case '^': /* ^ ^= */
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
563 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
564 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
565 t.kind = TOK.XorAssign;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
566 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
567 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
568 t.kind = TOK.Xor;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
569 goto Lcommon;
36
3c7210a722f7 - Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents: 35
diff changeset
570 case '%': /* % %= */
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
571 if (p[1] == '=') {
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
572 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
573 t.kind = TOK.ModAssign;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
574 }
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
575 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
576 t.kind = TOK.Mod;
29
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
577 goto Lcommon;
ef83eea26bbd - Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents: 28
diff changeset
578 // Single character tokens:
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
579 case '(':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
580 t.kind = TOK.LParen;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
581 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
582 case ')':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
583 t.kind = TOK.RParen;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
584 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
585 case '[':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
586 t.kind = TOK.LBracket;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
587 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
588 case ']':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
589 t.kind = TOK.RBracket;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
590 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
591 case '{':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
592 t.kind = TOK.LBrace;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
593 goto Lcommon;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
594 case '}':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
595 t.kind = TOK.RBrace;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
596 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
597 case ':':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
598 t.kind = TOK.Colon;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
599 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
600 case ';':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
601 t.kind = TOK.Semicolon;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
602 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
603 case '?':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
604 t.kind = TOK.Question;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
605 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
606 case ',':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
607 t.kind = TOK.Comma;
21
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
608 goto Lcommon;
c785c122e4e6 - Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents: 20
diff changeset
609 case '$':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
610 t.kind = TOK.Dollar;
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
611 Lcommon:
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
612 ++p;
26
c3d7373db241 - Added code for parsing Assign and Equal tokens.
aziz
parents: 25
diff changeset
613 Lcommon2:
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
614 t.end = p;
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
615 return;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
616 case '#':
360
b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence().
aziz
parents: 350
diff changeset
617 return scanSpecialTokenSequence(t);
20
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
618 default:
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
619 }
d6adfbd7c513 - Added code for parsing braces.
aziz
parents: 19
diff changeset
620
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
621 // Check for EOF
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
622 if (isEOF(c))
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
623 {
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
624 assert(isEOF(*p), ""~*p);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
625 t.kind = TOK.EOF;
411
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
626 t.end = p;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
627 tail = &t;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
628 assert(t.start == t.end);
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
629 return;
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
630 }
cca83c0c00fd Added __EOF__ token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 410
diff changeset
631
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
632 if (!isascii(c))
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
633 {
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
634 c = decodeUTF8();
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
635 if (isUniAlpha(c))
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
636 goto Lidentifier;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
637 }
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
638
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
639 error(t.start, MID.IllegalCharacter, cast(dchar)c);
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
640
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
641 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
642 t.kind = TOK.Illegal;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
643 t.setWhitespaceFlag();
390
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
644 t.dchar_ = c;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
645 t.end = p;
4d36eea1bbc9 Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 389
diff changeset
646 return;
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
647 }
3
4bbce78bfb1e - Added TOK enum.
aziz
parents: 2
diff changeset
648 }
4
92df59b1ec4a - Started implementation of scan().
aziz
parents: 3
diff changeset
649
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
650 /// Converts a string literal to an integer.
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
651 template toUint(char[] T)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
652 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
653 static assert(0 < T.length && T.length <= 4);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
654 static if (T.length == 1)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
655 const uint toUint = T[0];
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
656 else
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
657 const uint toUint = (T[0] << ((T.length-1)*8)) | toUint!(T[1..$]);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
658 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
659 static assert(toUint!("\xAA\xBB\xCC\xDD") == 0xAABBCCDD);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
660
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
661 /// Constructs case statements. E.g.:
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
662 /// ---
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
663 //// // case_!("<", "Less", "Lcommon") ->
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
664 /// case 60u:
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
665 /// t.kind = TOK.Less;
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
666 /// goto Lcommon;
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
667 /// ---
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
668 /// Note:Can't use this yet due to a $(DMDBUG 1534, bug) in DMD.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
669 template case_(char[] str, char[] kind, char[] label)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
670 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
671 const char[] case_ =
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
672 `case `~toUint!(str).stringof~`:`
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
673 `t.kind = TOK.`~kind~`;`
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
674 `goto `~label~`;`;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
675 }
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
676 //pragma(msg, case_!("<", "Less", "Lcommon"));
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
677
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
678 template case_L4(char[] str, TOK kind)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
679 {
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
680 const char[] case_L4 = case_!(str, kind, "Lcommon_4");
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
681 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
682
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
683 template case_L3(char[] str, TOK kind)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
684 {
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
685 const char[] case_L3 = case_!(str, kind, "Lcommon_3");
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
686 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
687
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
688 template case_L2(char[] str, TOK kind)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
689 {
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
690 const char[] case_L2 = case_!(str, kind, "Lcommon_2");
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
691 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
692
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
693 template case_L1(char[] str, TOK kind)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
694 {
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
695 const char[] case_L3 = case_!(str, kind, "Lcommon");
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
696 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
697
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
698 /// An alternative scan method.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
699 /// Profiling shows it's a bit slower.
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
700 public void scan_(ref Token t)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
701 in
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
702 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
703 assert(text.ptr <= p && p < end);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
704 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
705 out
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
706 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
707 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
708 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
709 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
710 body
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
711 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
712 // Scan whitespace.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
713 if (isspace(*p))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
714 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
715 t.ws = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
716 while (isspace(*++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
717 {}
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
718 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
719
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
720 // Scan a token.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
721 t.start = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
722 // Newline.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
723 switch (*p)
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
724 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
725 case '\r':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
726 if (p[1] == '\n')
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
727 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
728 case '\n':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
729 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
730 ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
731 ++lineNum;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
732 setLineBegin(p);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
733 // this.newline = &t;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
734 t.kind = TOK.Newline;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
735 t.setWhitespaceFlag();
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
736 t.newline.filePaths = this.filePaths;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
737 t.newline.oriLineNum = lineNum;
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
738 t.newline.setLineNum = lineNum_hline;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
739 t.end = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
740 return;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
741 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
742 if (isUnicodeNewline(p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
743 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
744 ++p; ++p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
745 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
746 }
418
7354f15cd5e9 Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 416
diff changeset
747 }
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
748
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
749 uint c = *p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
750 assert(end - p != 0);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
751 switch (end - p)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
752 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
753 case 1:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
754 goto L1character;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
755 case 2:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
756 c <<= 8; c |= p[1];
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
757 goto L2characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
758 case 3:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
759 c <<= 8; c |= p[1]; c <<= 8; c |= p[2];
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
760 goto L3characters;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
761 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
762 version(BigEndian)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
763 c = *cast(uint*)p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
764 else
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
765 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
766 c <<= 8; c |= p[1]; c <<= 8; c |= p[2]; c <<= 8; c |= p[3];
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
767 /+
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
768 c = *cast(uint*)p;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
769 asm
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
770 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
771 mov EDX, c;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
772 bswap EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
773 mov c, EDX;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
774 }
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
775 +/
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
776 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
777 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
778
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
779 // 4 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
780 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
781 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
782 case toUint!(">>>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
783 t.kind = TOK.RShiftAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
784 goto Lcommon_4;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
785 case toUint!("!<>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
786 t.kind = TOK.Unordered;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
787 Lcommon_4:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
788 p += 4;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
789 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
790 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
791 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
792 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
793
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
794 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
795 L3characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
796 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
797 // 3 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
798 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
799 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
800 case toUint!(">>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
801 t.kind = TOK.RShiftAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
802 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
803 case toUint!(">>>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
804 t.kind = TOK.URShift;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
805 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
806 case toUint!("<>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
807 t.kind = TOK.LorEorG;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
808 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
809 case toUint!("<<="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
810 t.kind = TOK.LShiftAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
811 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
812 case toUint!("!<="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
813 t.kind = TOK.UorG;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
814 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
815 case toUint!("!>="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
816 t.kind = TOK.UorL;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
817 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
818 case toUint!("!<>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
819 t.kind = TOK.UorE;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
820 goto Lcommon_3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
821 case toUint!("..."):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
822 t.kind = TOK.Ellipses;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
823 Lcommon_3:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
824 p += 3;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
825 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
826 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
827 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
828 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
829
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
830 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
831 L2characters:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
832 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
833 // 2 character tokens.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
834 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
835 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
836 case toUint!("/+"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
837 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
838 return scanNestedComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
839 case toUint!("/*"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
840 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
841 return scanBlockComment(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
842 case toUint!("//"):
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
843 ++p; // Skip /
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
844 assert(*p == '/');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
845 while (!isEndOfLine(++p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
846 isascii(*p) || decodeUTF8();
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
847 t.kind = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
848 t.setWhitespaceFlag();
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
849 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
850 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
851 case toUint!(">="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
852 t.kind = TOK.GreaterEqual;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
853 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
854 case toUint!(">>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
855 t.kind = TOK.RShift;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
856 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
857 case toUint!("<<"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
858 t.kind = TOK.LShift;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
859 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
860 case toUint!("<="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
861 t.kind = TOK.LessEqual;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
862 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
863 case toUint!("<>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
864 t.kind = TOK.LorG;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
865 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
866 case toUint!("!<"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
867 t.kind = TOK.UorGorE;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
868 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
869 case toUint!("!>"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
870 t.kind = TOK.UorLorE;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
871 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
872 case toUint!("!="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
873 t.kind = TOK.NotEqual;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
874 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
875 case toUint!(".."):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
876 t.kind = TOK.Slice;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
877 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
878 case toUint!("&&"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
879 t.kind = TOK.AndLogical;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
880 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
881 case toUint!("&="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
882 t.kind = TOK.AndAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
883 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
884 case toUint!("||"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
885 t.kind = TOK.OrLogical;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
886 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
887 case toUint!("|="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
888 t.kind = TOK.OrAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
889 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
890 case toUint!("++"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
891 t.kind = TOK.PlusPlus;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
892 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
893 case toUint!("+="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
894 t.kind = TOK.PlusAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
895 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
896 case toUint!("--"):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
897 t.kind = TOK.MinusMinus;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
898 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
899 case toUint!("-="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
900 t.kind = TOK.MinusAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
901 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
902 case toUint!("=="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
903 t.kind = TOK.Equal;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
904 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
905 case toUint!("~="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
906 t.kind = TOK.CatAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
907 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
908 case toUint!("*="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
909 t.kind = TOK.MulAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
910 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
911 case toUint!("/="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
912 t.kind = TOK.DivAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
913 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
914 case toUint!("^="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
915 t.kind = TOK.XorAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
916 goto Lcommon_2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
917 case toUint!("%="):
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
918 t.kind = TOK.ModAssign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
919 Lcommon_2:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
920 p += 2;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
921 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
922 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
923 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
924 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
925
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
926 c >>>= 8;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
927 L1character:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
928 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
929 assert(*p == c, Format("p={0},c={1}", *p, cast(dchar)c));
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
930 // 1 character tokens.
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
931 // TODO: consider storing the token type in ptable.
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
932 switch (c)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
933 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
934 case '\'':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
935 return scanCharacterLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
936 case '`':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
937 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
938 case '"':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
939 return scanNormalStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
940 case '\\':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
941 char[] buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
942 do
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
943 {
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
944 bool isBinary;
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
945 c = scanEscapeSequence(isBinary);
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
946 if (isascii(c) || isBinary)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
947 buffer ~= c;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
948 else
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
949 encodeUTF8(buffer, c);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
950 } while (*p == '\\')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
951 buffer ~= 0;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
952 t.kind = TOK.String;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
953 t.str = buffer;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
954 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
955 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
956 case '<':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
957 t.kind = TOK.Greater;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
958 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
959 case '>':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
960 t.kind = TOK.Less;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
961 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
962 case '^':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
963 t.kind = TOK.Xor;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
964 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
965 case '!':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
966 t.kind = TOK.Not;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
967 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
968 case '.':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
969 if (isdigit(p[1]))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
970 return scanReal(t);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
971 t.kind = TOK.Dot;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
972 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
973 case '&':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
974 t.kind = TOK.AndBinary;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
975 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
976 case '|':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
977 t.kind = TOK.OrBinary;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
978 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
979 case '+':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
980 t.kind = TOK.Plus;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
981 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
982 case '-':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
983 t.kind = TOK.Minus;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
984 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
985 case '=':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
986 t.kind = TOK.Assign;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
987 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
988 case '~':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
989 t.kind = TOK.Tilde;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
990 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
991 case '*':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
992 t.kind = TOK.Mul;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
993 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
994 case '/':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
995 t.kind = TOK.Div;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
996 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
997 case '%':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
998 t.kind = TOK.Mod;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
999 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1000 case '(':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1001 t.kind = TOK.LParen;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1002 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1003 case ')':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1004 t.kind = TOK.RParen;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1005 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1006 case '[':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1007 t.kind = TOK.LBracket;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1008 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1009 case ']':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1010 t.kind = TOK.RBracket;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1011 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1012 case '{':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1013 t.kind = TOK.LBrace;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1014 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1015 case '}':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1016 t.kind = TOK.RBrace;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1017 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1018 case ':':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1019 t.kind = TOK.Colon;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1020 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1021 case ';':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1022 t.kind = TOK.Semicolon;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1023 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1024 case '?':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1025 t.kind = TOK.Question;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1026 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1027 case ',':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1028 t.kind = TOK.Comma;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1029 goto Lcommon;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1030 case '$':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1031 t.kind = TOK.Dollar;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1032 Lcommon:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1033 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1034 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1035 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1036 case '#':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1037 return scanSpecialTokenSequence(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1038 default:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1039 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1040
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1041 assert(p == t.start);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1042 assert(*p == c);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1043
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1044 // TODO: consider moving isidbeg() and isdigit() up.
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1045 if (isidbeg(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1046 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1047 if (c == 'r' && p[1] == '"' && ++p)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1048 return scanRawStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1049 if (c == 'x' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1050 return scanHexStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1051 version(D2)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1052 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1053 if (c == 'q' && p[1] == '"')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1054 return scanDelimitedStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1055 if (c == 'q' && p[1] == '{')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1056 return scanTokenStringLiteral(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1057 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1058 // Scan identifier.
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1059 Lidentifier:
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1060 do
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1061 { c = *++p; }
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
1062 while (isident(c) || !isascii(c) && isUnicodeAlpha())
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1063
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1064 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1065
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
1066 auto id = IdTable.lookup(t.srcText);
769
5e3ef1b2011c Added and improved documentation.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 764
diff changeset
1067 t.kind = id.kind;
502
4e14cd1b24da Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 499
diff changeset
1068 t.ident = id;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1069
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1070 if (t.kind == TOK.Identifier || t.isKeyword)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1071 return;
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1072 else if (t.isSpecialToken)
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1073 finalizeSpecialToken(t);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1074 else if (t.kind == TOK.EOF)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1075 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1076 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1077 assert(t.srcText == "__EOF__");
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1078 }
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1079 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1080 assert(0, "unexpected token type: " ~ Token.toString(t.kind));
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1081 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1082 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1083
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1084 if (isdigit(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1085 return scanNumber(t);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1086
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1087 // Check for EOF
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1088 if (isEOF(c))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1089 {
497
0ffcc4ff82f3 Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
1090 assert(isEOF(*p), *p~"");
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1091 t.kind = TOK.EOF;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1092 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1093 tail = &t;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1094 assert(t.start == t.end);
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1095 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1096 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1097
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1098 if (!isascii(c))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1099 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1100 c = decodeUTF8();
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1101 if (isUniAlpha(c))
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1102 goto Lidentifier;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1103 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1104
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1105 error(t.start, MID.IllegalCharacter, cast(dchar)c);
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1106
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1107 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1108 t.kind = TOK.Illegal;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
1109 t.setWhitespaceFlag();
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1110 t.dchar_ = c;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1111 t.end = p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1112 return;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1113 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1114
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1115 /// Scans a block comment.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1116 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1117 /// BlockComment := "/*" AnyChar* "*/"
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1118 void scanBlockComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1119 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1120 assert(p[-1] == '/' && *p == '*');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1121 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1122 auto tokenLineBegin = lineBegin;
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1123 Loop:
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1124 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1125 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1126 switch (*++p)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1127 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1128 case '*':
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1129 if (p[1] != '/')
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1130 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1131 p += 2;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1132 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1133 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1134 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1135 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1136 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1137 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1138 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1139 setLineBegin(p+1);
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1140 break;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1141 default:
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1142 if (!isascii(*p))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1143 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1144 if (isUnicodeNewlineChar(decodeUTF8()))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1145 goto case '\n';
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1146 }
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1147 else if (isEOF(*p))
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1148 {
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1149 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedBlockComment);
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1150 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1151 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1152 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1153 }
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1154 t.kind = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
1155 t.setWhitespaceFlag();
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1156 t.end = p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1157 return;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1158 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1159
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1160 /// Scans a nested comment.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1161 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1162 /// NestedComment := "/+" (AnyChar* | NestedComment) "+/"
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1163 void scanNestedComment(ref Token t)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1164 {
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1165 assert(p[-1] == '/' && *p == '+');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1166 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1167 auto tokenLineBegin = lineBegin;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1168 uint level = 1;
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1169 Loop:
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1170 while (1)
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1171 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1172 switch (*++p)
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1173 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1174 case '/':
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1175 if (p[1] == '+')
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1176 ++p, ++level;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1177 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1178 case '+':
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1179 if (p[1] != '/')
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1180 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1181 ++p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1182 if (--level != 0)
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1183 continue;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1184 ++p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1185 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1186 case '\r':
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1187 if (p[1] == '\n')
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1188 ++p;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1189 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1190 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1191 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1192 setLineBegin(p+1);
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1193 continue;
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1194 default:
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1195 if (!isascii(*p))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1196 {
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1197 if (isUnicodeNewlineChar(decodeUTF8()))
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1198 goto case '\n';
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1199 }
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1200 else if (isEOF(*p))
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1201 {
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1202 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedNestedComment);
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1203 break Loop;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1204 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1205 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1206 }
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1207 t.kind = TOK.Comment;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
1208 t.setWhitespaceFlag();
498
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1209 t.end = p;
49c201b5c465 Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 497
diff changeset
1210 return;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1211 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
1212
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1213 /// Scans the postfix character of a string literal.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1214 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1215 /// PostfixChar := "c" | "w" | "d"
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1216 char scanPostfix()
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1217 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1218 assert(p[-1] == '"' || p[-1] == '`' ||
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1219 { version(D2) return p[-1] == '}';
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1220 else return 0; }()
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1221 );
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1222 switch (*p)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1223 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1224 case 'c':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1225 case 'w':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1226 case 'd':
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1227 return *p++;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1228 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1229 return 0;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1230 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1231 assert(0);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1232 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1233
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1234 /// Scans a normal string literal.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1235 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1236 /// NormalStringLiteral := "\"" Char* "\""
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1237 void scanNormalStringLiteral(ref Token t)
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1238 {
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1239 assert(*p == '"');
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1240 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1241 auto tokenLineBegin = lineBegin;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1242 t.kind = TOK.String;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1243 char[] buffer;
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1244 uint c;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1245 while (1)
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1246 {
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1247 c = *++p;
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1248 switch (c)
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1249 {
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1250 case '"':
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1251 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1252 t.pf = scanPostfix();
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1253 Lreturn:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1254 t.str = buffer ~ '\0';
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1255 t.end = p;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1256 return;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1257 case '\\':
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1258 bool isBinary;
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1259 c = scanEscapeSequence(isBinary);
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1260 --p;
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1261 if (isascii(c) || isBinary)
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1262 buffer ~= c;
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1263 else
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1264 encodeUTF8(buffer, c);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1265 continue;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1266 case '\r':
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1267 if (p[1] == '\n')
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1268 ++p;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1269 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1270 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1271 c = '\n'; // Convert Newline to \n.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1272 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1273 setLineBegin(p+1);
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1274 break;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1275 case 0, _Z_:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1276 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedString);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1277 goto Lreturn;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1278 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1279 if (!isascii(c))
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1280 {
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1281 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1282 if (isUnicodeNewlineChar(c))
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1283 goto case '\n';
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1284 encodeUTF8(buffer, c);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1285 continue;
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1286 }
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1287 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1288 assert(isascii(c));
384
f7ce725e79c3 Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 383
diff changeset
1289 buffer ~= c;
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1290 }
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
1291 assert(0);
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1292 }
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
1293
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1294 /// Scans a character literal.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1295 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1296 /// CharLiteral := "'" Char "'"
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1297 void scanCharacterLiteral(ref Token t)
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1298 {
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1299 assert(*p == '\'');
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1300 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1301 t.kind = TOK.CharLiteral;
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1302 switch (*p)
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1303 {
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1304 case '\\':
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1305 bool notused;
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1306 t.dchar_ = scanEscapeSequence(notused);
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1307 break;
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1308 case '\'':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1309 error(t.start, MID.EmptyCharacterLiteral);
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1310 break;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1311 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1312 if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1313 break;
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1314 uint c = *p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1315 if (!isascii(c))
49
7f0fa15dcffc - Renamed function.
aziz
parents: 48
diff changeset
1316 c = decodeUTF8();
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1317 t.dchar_ = c;
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1318 ++p;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1319 }
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1320
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1321 if (*p == '\'')
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1322 ++p;
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1323 else
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1324 error(t.start, MID.UnterminatedCharacterLiteral);
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1325 t.end = p;
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1326 }
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
1327
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1328 /// Scans a raw string literal.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1329 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1330 /// RawStringLiteral := "r\"" AnyChar* "\"" | "`" AnyChar* "`"
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1331 void scanRawStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1332 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1333 assert(*p == '`' || *p == '"' && p[-1] == 'r');
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1334 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1335 auto tokenLineBegin = lineBegin;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1336 t.kind = TOK.String;
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1337 uint delim = *p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1338 char[] buffer;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1339 uint c;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1340 while (1)
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1341 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1342 c = *++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1343 switch (c)
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1344 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1345 case '\r':
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1346 if (p[1] == '\n')
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1347 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1348 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1349 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1350 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1351 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1352 setLineBegin(p+1);
52
f65a83c27638 - Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well.
aziz
parents: 51
diff changeset
1353 break;
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1354 case '`':
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1355 case '"':
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1356 if (c == delim)
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1357 {
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1358 ++p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1359 t.pf = scanPostfix();
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1360 Lreturn:
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1361 t.str = buffer ~ '\0';
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1362 t.end = p;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1363 return;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1364 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1365 break;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1366 case 0, _Z_:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1367 error(tokenLineNum, tokenLineBegin, t.start,
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1368 delim == 'r' ? MID.UnterminatedRawString : MID.UnterminatedBackQuoteString);
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1369 goto Lreturn;
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1370 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1371 if (!isascii(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1372 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1373 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1374 if (isUnicodeNewlineChar(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1375 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1376 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1377 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1378 }
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1379 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1380 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1381 buffer ~= c;
33
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1382 }
cf3047cf3cd2 - Added code for parsing back quote and raw strings.
aziz
parents: 32
diff changeset
1383 assert(0);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1384 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1385
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1386 /// Scans a hexadecimal string literal.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1387 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1388 /// HexStringLiteral := "x\"" (HexChar HexChar)* "\""
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1389 void scanHexStringLiteral(ref Token t)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1390 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1391 assert(p[0] == 'x' && p[1] == '"');
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1392 t.kind = TOK.String;
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1393
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1394 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1395 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1396
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1397 uint c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1398 ubyte[] buffer;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1399 ubyte h; // hex number
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1400 uint n; // number of hex digits
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1401
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1402 ++p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1403 assert(*p == '"');
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1404 while (1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1405 {
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1406 c = *++p;
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1407 switch (c)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1408 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1409 case '"':
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1410 if (n & 1)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1411 error(tokenLineNum, tokenLineBegin, t.start, MID.OddNumberOfDigitsInHexString);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1412 ++p;
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1413 t.pf = scanPostfix();
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1414 Lreturn:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1415 t.str = cast(string) (buffer ~= 0);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1416 t.end = p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1417 return;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1418 case '\r':
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1419 if (p[1] == '\n')
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1420 ++p;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1421 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1422 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1423 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1424 setLineBegin(p+1);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1425 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1426 default:
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1427 if (ishexad(c))
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1428 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1429 if (c <= '9')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1430 c -= '0';
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1431 else if (c <= 'F')
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1432 c -= 'A' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1433 else
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1434 c -= 'a' - 10;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1435
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1436 if (n & 1)
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1437 {
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1438 h <<= 4;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1439 h |= c;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1440 buffer ~= h;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1441 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1442 else
112
004d98df65af - Implemented parseInterfaceDeclaration().
aziz
parents: 105
diff changeset
1443 h = cast(ubyte)c;
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1444 ++n;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1445 continue;
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1446 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1447 else if (isspace(c))
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
1448 continue; // Skip spaces.
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
1449 else if (isEOF(c))
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1450 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1451 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedHexString);
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1452 t.pf = 0;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1453 goto Lreturn;
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1454 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1455 else
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1456 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1457 auto errorAt = p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1458 if (!isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1459 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1460 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1461 if (isUnicodeNewlineChar(c))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1462 goto case '\n';
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1463 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1464 error(errorAt, MID.NonHexCharInHexString, cast(dchar)c);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1465 }
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1466 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1467 }
53
1786c2825491 - Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents: 52
diff changeset
1468 assert(0);
31
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1469 }
94f09f4e988e - Added struct for strings to Token with 'pf' = postfix.
aziz
parents: 30
diff changeset
1470
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1471 version(DDoc)
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1472 {
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1473 /// Scans a delimited string literal.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1474 void scanDelimitedStringLiteral(ref Token t);
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1475 /// Scans a token string literal.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1476 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1477 /// TokenStringLiteral := "q{" Token* "}"
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1478 void scanTokenStringLiteral(ref Token t);
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1479 }
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1480 else
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1481 version(D2)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1482 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1483 void scanDelimitedStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1484 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1485 assert(p[0] == 'q' && p[1] == '"');
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1486 t.kind = TOK.String;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1487
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1488 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1489 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1490
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1491 char[] buffer;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1492 dchar opening_delim = 0, // 0 if no nested delimiter or '[', '(', '<', '{'
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1493 closing_delim; // Will be ']', ')', '>', '},
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1494 // the first character of an identifier or
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1495 // any other Unicode/ASCII character.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1496 char[] str_delim; // Identifier delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1497 uint level = 1; // Counter for nestable delimiters.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1498
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1499 ++p; ++p; // Skip q"
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1500 uint c = *p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1501 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1502 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1503 case '(':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1504 opening_delim = c;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1505 closing_delim = ')'; // c + 1
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1506 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1507 case '[', '<', '{':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1508 opening_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1509 closing_delim = c + 2; // Get to closing counterpart. Feature of ASCII table.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1510 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1511 default:
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1512 dchar scanNewline()
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1513 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1514 switch (*p)
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1515 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1516 case '\r':
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1517 if (p[1] == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1518 ++p;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1519 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1520 assert(isNewlineEnd(p));
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1521 ++p;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1522 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1523 setLineBegin(p);
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1524 return '\n';
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1525 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1526 if (isUnicodeNewline(p))
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1527 {
419
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
1528 ++p; ++p;
89e40d43065d Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 418
diff changeset
1529 goto case '\n';
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1530 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1531 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1532 return 0;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1533 }
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1534 // Skip leading newlines:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1535 while (scanNewline() != 0)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1536 {}
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1537 assert(!isNewline(p));
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1538
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1539 char* begin = p;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1540 c = *p;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1541 closing_delim = c;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1542 // TODO: Check for non-printable characters?
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1543 if (!isascii(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1544 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1545 closing_delim = decodeUTF8();
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1546 if (!isUniAlpha(closing_delim))
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1547 break; // Not an identifier.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1548 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1549 else if (!isidbeg(c))
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1550 break; // Not an identifier.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1551
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1552 // Parse Identifier + EndOfLine
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1553 do
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1554 { c = *++p; }
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
1555 while (isident(c) || !isascii(c) && isUnicodeAlpha())
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1556 // Store identifier
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1557 str_delim = begin[0..p-begin];
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1558 // Scan newline
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1559 if (scanNewline() == '\n')
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1560 --p; // Go back one because of "c = *++p;" in main loop.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1561 else
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1562 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1563 // TODO: error(p, MID.ExpectedNewlineAfterIdentDelim);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1564 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1565 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1566
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1567 bool checkStringDelim(char* p)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1568 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1569 assert(str_delim.length != 0);
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1570 if (buffer[$-1] == '\n' && // Last character copied to buffer must be '\n'.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1571 end-p >= str_delim.length && // Check remaining length.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1572 p[0..str_delim.length] == str_delim) // Compare.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1573 return true;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1574 return false;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1575 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1576
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1577 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1578 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1579 c = *++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1580 switch (c)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1581 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1582 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1583 if (p[1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1584 ++p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1585 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1586 assert(isNewlineEnd(p));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1587 c = '\n'; // Convert Newline to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1588 ++lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1589 setLineBegin(p+1);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1590 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1591 case 0, _Z_:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1592 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedDelimitedString);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1593 goto Lreturn3;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1594 default:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1595 if (!isascii(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1596 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1597 auto begin = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1598 c = decodeUTF8();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1599 if (isUnicodeNewlineChar(c))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1600 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1601 if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1602 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1603 if (str_delim.length)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1604 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1605 if (checkStringDelim(begin))
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1606 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1607 p = begin + str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1608 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1609 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1610 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1611 else
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1612 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1613 assert(level == 1);
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1614 --level;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1615 goto Lreturn;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1616 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1617 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1618 encodeUTF8(buffer, c);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1619 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1620 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1621 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1622 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1623 if (c == opening_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1624 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1625 else if (c == closing_delim)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1626 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1627 if (str_delim.length)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1628 {
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1629 if (checkStringDelim(p))
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1630 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1631 p += str_delim.length;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1632 goto Lreturn2;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1633 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1634 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1635 else if (--level == 0)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1636 goto Lreturn;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1637 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1638 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1639 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1640 assert(isascii(c));
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1641 buffer ~= c;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1642 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1643 Lreturn: // Character delimiter.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1644 assert(c == closing_delim);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1645 assert(level == 0);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1646 ++p; // Skip closing delimiter.
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1647 Lreturn2: // String delimiter.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1648 if (*p == '"')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1649 ++p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1650 else
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1651 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1652 // TODO: error(p, MID.ExpectedDblQuoteAfterDelim, str_delim.length ? str_delim : closing_delim~"");
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1653 }
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1654
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1655 t.pf = scanPostfix();
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1656 Lreturn3: // Error.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1657 t.str = buffer ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1658 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1659 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1660
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1661 void scanTokenStringLiteral(ref Token t)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1662 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1663 assert(p[0] == 'q' && p[1] == '{');
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1664 t.kind = TOK.String;
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1665
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1666 auto tokenLineNum = lineNum;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1667 auto tokenLineBegin = lineBegin;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1668
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1669 // A guard against changes to particular members:
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1670 // this.lineNum_hline and this.errorPath
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1671 ++inTokenString;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1672
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1673 uint lineNum = this.lineNum;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1674 uint level = 1;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1675
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1676 ++p; ++p; // Skip q{
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1677
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1678 auto prev_t = &t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1679 Token* token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1680 while (1)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1681 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1682 token = new Token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1683 scan(*token);
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1684 // Save the tokens in a doubly linked list.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1685 // Could be useful for various tools.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1686 token.prev = prev_t;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1687 prev_t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1688 prev_t = token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1689 switch (token.kind)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1690 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1691 case TOK.LBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1692 ++level;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1693 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1694 case TOK.RBrace:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1695 if (--level == 0)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1696 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1697 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1698 t.next = null;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1699 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1700 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1701 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1702 case TOK.EOF:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1703 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedTokenString);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1704 t.tok_str = t.next;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1705 t.next = token;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1706 break;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1707 default:
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1708 continue;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1709 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1710 break; // Exit loop.
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1711 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1712
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1713 assert(token.kind == TOK.RBrace || token.kind == TOK.EOF);
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1714 assert(token.kind == TOK.RBrace && t.next is null ||
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1715 token.kind == TOK.EOF && t.next !is null);
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1716
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1717 char[] buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1718 // token points to } or EOF
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
1719 if (token.kind == TOK.EOF)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1720 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1721 t.end = token.start;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1722 buffer = t.srcText[2..$].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1723 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1724 else
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1725 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1726 // Assign to buffer before scanPostfix().
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1727 t.end = p;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1728 buffer = t.srcText[2..$-1].dup ~ '\0';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1729 t.pf = scanPostfix();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1730 t.end = p; // Assign again because of postfix.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1731 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1732 // Convert newlines to '\n'.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1733 if (lineNum != this.lineNum)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1734 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1735 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1736 uint i, j;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1737 for (; i < buffer.length; ++i)
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1738 switch (buffer[i])
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1739 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1740 case '\r':
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1741 if (buffer[i+1] == '\n')
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1742 ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1743 case '\n':
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1744 assert(isNewlineEnd(buffer.ptr + i));
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1745 buffer[j++] = '\n'; // Convert Newline to '\n'.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1746 break;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1747 default:
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1748 if (isUnicodeNewline(buffer.ptr + i))
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1749 {
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1750 ++i; ++i;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1751 goto case '\n';
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1752 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1753 buffer[j++] = buffer[i]; // Copy.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1754 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1755 buffer.length = j; // Adjust length.
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1756 }
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1757 assert(buffer[$-1] == '\0');
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1758 t.str = buffer;
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1759
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1760 --inTokenString;
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1761 }
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1762 } // version(D2)
383
6a5fc22cae34 Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 382
diff changeset
1763
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1764 /// Scans an escape sequence.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1765 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1766 /// EscapeSequence := "\" (Octal{1,3} | ("x" Hex{2}) |
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1767 /// ("u" Hex{4}) | ("U" Hex{8}) |
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1768 /// "'" | "\"" | "\\" | "?" | "a" |
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1769 /// "b" | "f" | "n" | "r" | "t" | "v")
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1770 /// Params:
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1771 /// isBinary = set to true for octal and hexadecimal escapes.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1772 /// Returns: the escape value.
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1773 dchar scanEscapeSequence(ref bool isBinary)
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1774 out(result)
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
1775 { assert(isValidChar(result)); }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1776 body
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1777 {
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1778 assert(*p == '\\');
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1779
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1780 auto sequenceStart = p; // Used for error reporting.
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1781
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1782 ++p;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1783 uint c = char2ev(*p);
82
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1784 if (c)
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1785 {
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1786 ++p;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1787 return c;
46
e0dbd4722f74 - Scanning character literals correctly now.
aziz
parents: 45
diff changeset
1788 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1789
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1790 uint digits = 2;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1791
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1792 switch (*p)
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1793 {
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1794 case 'x':
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1795 isBinary = true;
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1796 case_Unicode:
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1797 assert(c == 0);
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1798 assert(digits == 2 || digits == 4 || digits == 8);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1799 while (1)
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1800 {
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1801 ++p;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1802 if (ishexad(*p))
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1803 {
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1804 c *= 16;
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1805 if (*p <= '9')
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1806 c += *p - '0';
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1807 else if (*p <= 'F')
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1808 c += *p - 'A' + 10;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1809 else
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1810 c += *p - 'a' + 10;
82
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1811
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1812 if (--digits == 0)
82
fc645fb2fe72 - scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents: 71
diff changeset
1813 {
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1814 ++p;
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
1815 if (isValidChar(c))
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1816 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1817
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1818 error(sequenceStart, MID.InvalidUnicodeEscapeSequence,
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1819 sequenceStart[0..p-sequenceStart]);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1820 break;
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1821 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1822 continue;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1823 }
432
3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 430
diff changeset
1824
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1825 error(sequenceStart, MID.InsufficientHexDigits,
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1826 sequenceStart[0..p-sequenceStart]);
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1827 break;
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
1828 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1829 break;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1830 case 'u':
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1831 digits = 4;
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1832 goto case_Unicode;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1833 case 'U':
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1834 digits = 8;
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1835 goto case_Unicode;
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1836 default:
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1837 if (isoctal(*p))
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1838 {
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
1839 isBinary = true;
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1840 assert(c == 0);
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1841 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1842 ++p;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1843 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1844 return c;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1845 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1846 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1847 ++p;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1848 if (!isoctal(*p))
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1849 return c;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1850 c *= 8;
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1851 c += *p - '0';
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1852 ++p;
803
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
1853 if (c > 0xFF)
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
1854 error(sequenceStart, MSG.InvalidOctalEscapeSequence,
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
1855 sequenceStart[0..p-sequenceStart]);
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
1856 return c; // Return valid escape value.
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1857 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1858 else if(*p == '&')
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1859 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1860 if (isalpha(*++p))
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1861 {
272
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1862 auto begin = p;
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1863 while (isalnum(*++p))
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1864 {}
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1865
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1866 if (*p == ';')
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1867 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1868 // Pass entity excluding '&' and ';'.
272
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1869 c = entity2Unicode(begin[0..p - begin]);
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1870 ++p; // Skip ;
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1871 if (c != 0xFFFF)
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1872 return c; // Return valid escape value.
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1873 else
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1874 error(sequenceStart, MID.UndefinedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
272
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1875 }
0bde32503976 - Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents: 249
diff changeset
1876 else
432
3ead178e0662 Added message MID.InvalidUnicodeEscapeSequence.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 430
diff changeset
1877 error(sequenceStart, MID.UnterminatedHTMLEntity, sequenceStart[0 .. p - sequenceStart]);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1878 }
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
1879 else
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1880 error(sequenceStart, MID.InvalidBeginHTMLEntity);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1881 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1882 else if (isEndOfLine(p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1883 error(sequenceStart, MID.UndefinedEscapeSequence,
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
1884 isEOF(*p) ? `\EOF` : `\NewLine`);
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1885 else
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1886 {
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1887 char[] str = `\`;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1888 if (isascii(c))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1889 str ~= *p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
1890 else
387
ad0cbd1c8881 Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 386
diff changeset
1891 encodeUTF8(str, decodeUTF8());
ad0cbd1c8881 Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 386
diff changeset
1892 ++p;
ad0cbd1c8881 Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 386
diff changeset
1893 // TODO: check for unprintable character?
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1894 error(sequenceStart, MID.UndefinedEscapeSequence, str);
386
392a0068fc61 Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 384
diff changeset
1895 }
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1896 }
433
063cd016c913 Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 432
diff changeset
1897 return REPLACEMENT_CHAR; // Error: return replacement character.
45
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1898 }
cc84b9ca9d0a - Implemented escape sequence scanner.
aziz
parents: 44
diff changeset
1899
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1900 /// Scans a number literal.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1901 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1902 /// $(PRE
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1903 /// IntegerLiteral := (Dec|Hex|Bin|Oct)Suffix?
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1904 /// Dec := (0|[1-9][0-9_]*)
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1905 /// Hex := 0[xX][_]*[0-9a-zA-Z][0-9a-zA-Z_]*
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1906 /// Bin := 0[bB][_]*[01][01_]*
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1907 /// Oct := 0[0-7_]*
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1908 /// Suffix := (L[uU]?|[uU]L?)
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1909 /// )
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
1910 /// Invalid: "0b_", "0x_", "._" etc.
15
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
1911 void scanNumber(ref Token t)
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
1912 {
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1913 ulong ulong_;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1914 bool overflow;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1915 bool isDecimal;
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
1916 size_t digits;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1917
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1918 if (*p != '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1919 goto LscanInteger;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1920 ++p; // skip zero
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1921 // check for xX bB ...
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1922 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1923 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1924 case 'x','X':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1925 goto LscanHex;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1926 case 'b','B':
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1927 goto LscanBinary;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1928 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1929 if (p[1] == 'i')
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1930 goto LscanReal; // 0Li
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1931 break; // 0L
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1932 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1933 if (p[1] == '.')
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1934 break; // 0..
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1935 // 0.
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1936 case 'i','f','F', // Imaginary and float literal suffixes.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
1937 'e', 'E': // Float exponent.
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1938 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1939 default:
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1940 if (*p == '_')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1941 goto LscanOctal; // 0_
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1942 else if (isdigit(*p))
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1943 {
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1944 if (*p == '8' || *p == '9')
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1945 goto Loctal_hasDecimalDigits; // 08 or 09
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1946 else
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1947 goto Loctal_enter_loop; // 0[0-7]
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1948 }
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1949 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1950
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1951 // Number 0
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1952 assert(p[-1] == '0');
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
1953 assert(*p != '_' && !isdigit(*p));
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1954 assert(ulong_ == 0);
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1955 isDecimal = true;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1956 goto Lfinalize;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1957
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1958 LscanInteger:
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1959 assert(*p != 0 && isdigit(*p));
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1960 isDecimal = true;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1961 goto Lenter_loop_int;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1962 while (1)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1963 {
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1964 if (*++p == '_')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1965 continue;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1966 if (!isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
1967 break;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
1968 Lenter_loop_int:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1969 if (ulong_ < ulong.max/10 || (ulong_ == ulong.max/10 && *p <= '5'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1970 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1971 ulong_ *= 10;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1972 ulong_ += *p - '0';
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1973 continue;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1974 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1975 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1976 overflow = true;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1977 while (isdigit(*++p)) {}
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1978 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1979 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1980
61
512cd2248dfc - Fix: issueing error on hexadecimal number overflow.
aziz
parents: 60
diff changeset
1981 // The number could be a float, so check overflow below.
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1982 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1983 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1984 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1985 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1986 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1987 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1988 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1989 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1990 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1991 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1992 goto LscanReal;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1993 default:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1994 }
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1995
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1996 if (overflow)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
1997 error(t.start, MID.OverflowDecimalNumber);
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
1998
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
1999 assert((isdigit(p[-1]) || p[-1] == '_') && !isdigit(*p) && *p != '_');
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2000 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2001
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2002 LscanHex:
61
512cd2248dfc - Fix: issueing error on hexadecimal number overflow.
aziz
parents: 60
diff changeset
2003 assert(digits == 0);
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
2004 assert(*p == 'x' || *p == 'X');
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2005 while (1)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2006 {
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2007 if (*++p == '_')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2008 continue;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2009 if (!ishexad(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2010 break;
61
512cd2248dfc - Fix: issueing error on hexadecimal number overflow.
aziz
parents: 60
diff changeset
2011 ++digits;
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
2012 ulong_ *= 16;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
2013 if (*p <= '9')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
2014 ulong_ += *p - '0';
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
2015 else if (*p <= 'F')
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
2016 ulong_ += *p - 'A' + 10;
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
2017 else
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
2018 ulong_ += *p - 'a' + 10;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2019 }
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
2020
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
2021 assert(ishexad(p[-1]) || p[-1] == '_' || p[-1] == 'x' || p[-1] == 'X');
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2022 assert(!ishexad(*p) && *p != '_');
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2023
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2024 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2025 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2026 case '.':
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2027 if (p[1] == '.')
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2028 break;
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2029 case 'p', 'P':
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2030 return scanHexReal(t);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2031 default:
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2032 }
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2033
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2034 if (digits == 0 || digits > 16)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2035 error(t.start, digits == 0 ? MID.NoDigitsInHexNumber : MID.OverflowHexNumber);
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2036
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2037 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2038
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2039 LscanBinary:
57
c0f1c8be3a47 - Added code for converting hex characters to binary numbers.
aziz
parents: 56
diff changeset
2040 assert(digits == 0);
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
2041 assert(*p == 'b' || *p == 'B');
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2042 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2043 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2044 if (*++p == '0')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2045 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2046 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2047 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2048 }
367
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
2049 else if (*p == '1')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2050 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2051 ++digits;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2052 ulong_ *= 2;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2053 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2054 }
367
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
2055 else if (*p == '_')
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
2056 continue;
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
2057 else
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
2058 break;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2059 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2060
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2061 if (digits == 0 || digits > 64)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2062 error(t.start, digits == 0 ? MID.NoDigitsInBinNumber : MID.OverflowBinaryNumber);
59
3e594725899a - Issuing error when no digits were found in hex and binary numbers.
aziz
parents: 58
diff changeset
2063
410
4d9ee8e60712 Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 403
diff changeset
2064 assert(p[-1] == '0' || p[-1] == '1' || p[-1] == '_' || p[-1] == 'b' || p[-1] == 'B', p[-1] ~ "");
367
dda55fae37de - ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents: 365
diff changeset
2065 assert( !(*p == '0' || *p == '1' || *p == '_') );
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2066 goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2067
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2068 LscanOctal:
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2069 assert(*p == '_');
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2070 while (1)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2071 {
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2072 if (*++p == '_')
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2073 continue;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2074 if (!isoctal(*p))
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2075 break;
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2076 Loctal_enter_loop:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2077 if (ulong_ < ulong.max/2 || (ulong_ == ulong.max/2 && *p <= '1'))
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2078 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2079 ulong_ *= 8;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2080 ulong_ += *p - '0';
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2081 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2082 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2083 // Overflow: skip following digits.
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2084 overflow = true;
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2085 while (isoctal(*++p)) {}
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2086 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2087 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2088
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2089 bool hasDecimalDigits;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2090 if (isdigit(*p))
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2091 {
396
0a4619735ce9 Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 393
diff changeset
2092 Loctal_hasDecimalDigits:
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2093 hasDecimalDigits = true;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2094 while (isdigit(*++p)) {}
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2095 }
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2096
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2097 // The number could be a float, so check errors below.
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2098 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2099 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2100 case '.':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2101 if (p[1] != '.')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2102 goto LscanReal;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2103 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2104 case 'L':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2105 if (p[1] != 'i')
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2106 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2107 case 'i', 'f', 'F', 'e', 'E':
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2108 goto LscanReal;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2109 default:
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2110 }
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2111
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2112 if (hasDecimalDigits)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2113 error(t.start, MID.OctalNumberHasDecimals);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2114
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2115 if (overflow)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2116 error(t.start, MID.OverflowOctalNumber);
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2117 // goto Lfinalize;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2118
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2119 Lfinalize:
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2120 enum Suffix
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2121 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2122 None = 0,
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2123 Unsigned = 1,
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2124 Long = 2
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2125 }
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2126
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2127 // Scan optional suffix: L, Lu, LU, u, uL, U or UL.
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2128 Suffix suffix;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2129 while (1)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2130 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2131 switch (*p)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2132 {
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2133 case 'L':
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2134 if (suffix & Suffix.Long)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2135 break;
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2136 suffix |= Suffix.Long;
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2137 ++p;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2138 continue;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2139 case 'u', 'U':
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2140 if (suffix & Suffix.Unsigned)
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2141 break;
60
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2142 suffix |= Suffix.Unsigned;
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2143 ++p;
32cc23bd217b - Fixed number suffix scanning.
aziz
parents: 59
diff changeset
2144 continue;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2145 default:
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2146 break;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2147 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2148 break;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2149 }
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2150
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2151 // Determine type of Integer.
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2152 switch (suffix)
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2153 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2154 case Suffix.None:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2155 if (ulong_ & 0x8000_0000_0000_0000)
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2156 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2157 if (isDecimal)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2158 error(t.start, MID.OverflowDecimalSign);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2159 t.kind = TOK.Uint64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2160 }
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2161 else if (ulong_ & 0xFFFF_FFFF_0000_0000)
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2162 t.kind = TOK.Int64;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2163 else if (ulong_ & 0x8000_0000)
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2164 t.kind = isDecimal ? TOK.Int64 : TOK.Uint32;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2165 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2166 t.kind = TOK.Int32;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2167 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2168 case Suffix.Unsigned:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2169 if (ulong_ & 0xFFFF_FFFF_0000_0000)
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2170 t.kind = TOK.Uint64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2171 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2172 t.kind = TOK.Uint32;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2173 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2174 case Suffix.Long:
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2175 if (ulong_ & 0x8000_0000_0000_0000)
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2176 {
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2177 if (isDecimal)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2178 error(t.start, MID.OverflowDecimalSign);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2179 t.kind = TOK.Uint64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2180 }
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2181 else
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2182 t.kind = TOK.Int64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2183 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2184 case Suffix.Unsigned | Suffix.Long:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2185 t.kind = TOK.Uint64;
58
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2186 break;
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2187 default:
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2188 assert(0);
50bb7fc9db44 - The types of integers are recognized now.
aziz
parents: 57
diff changeset
2189 }
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2190 t.ulong_ = ulong_;
15
c70c028e47dd - Started implementation of lexing numbers.
aziz
parents: 14
diff changeset
2191 t.end = p;
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2192 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2193 LscanReal:
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2194 scanReal(t);
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2195 return;
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2196 }
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2197
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2198 /// Scans a floating point number literal.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2199 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2200 /// $(PRE
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2201 /// FloatLiteral := Float[fFL]?i?
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2202 /// Float := DecFloat | HexFloat
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2203 /// DecFloat := ([0-9][0-9_]*[.][0-9_]*DecExponent?) |
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2204 /// [.][0-9][0-9_]*DecExponent? | [0-9][0-9_]*DecExponent
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2205 /// DecExponent := [eE][+-]?[0-9][0-9_]*
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2206 /// HexFloat := 0[xX](HexDigits[.]HexDigits |
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2207 /// [.][0-9a-zA-Z]HexDigits? |
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2208 /// HexDigits)HexExponent
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2209 /// HexExponent := [pP][+-]?[0-9][0-9_]*
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2210 /// )
56
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2211 void scanReal(ref Token t)
63af7ddf52e1 - Started properly implementing number scanner. Added stub for real numbers.
aziz
parents: 55
diff changeset
2212 {
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2213 if (*p == '.')
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2214 {
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2215 assert(p[1] != '.');
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2216 // This function was called by scan() or scanNumber().
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2217 while (isdigit(*++p) || *p == '_') {}
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2218 }
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2219 else
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2220 // This function was called by scanNumber().
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2221 assert(delegate ()
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2222 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2223 switch (*p)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2224 {
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2225 case 'L':
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2226 if (p[1] != 'i')
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2227 return false;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2228 case 'i', 'f', 'F', 'e', 'E':
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2229 return true;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2230 default:
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2231 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2232 return false;
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2233 }()
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2234 );
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2235
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2236 // Scan exponent.
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2237 if (*p == 'e' || *p == 'E')
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2238 {
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2239 ++p;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2240 if (*p == '-' || *p == '+')
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2241 ++p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2242 if (isdigit(*p))
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2243 while (isdigit(*++p) || *p == '_') {}
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2244 else
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2245 error(t.start, MID.FloatExpMustStartWithDigit);
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2246 }
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2247
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2248 // Copy whole number and remove underscores from buffer.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2249 char[] buffer = t.start[0..p-t.start].dup;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2250 uint j;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2251 foreach (c; buffer)
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2252 if (c != '_')
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2253 buffer[j++] = c;
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2254 buffer.length = j; // Adjust length.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2255 buffer ~= 0; // Terminate for C functions.
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2256
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2257 finalizeFloat(t, buffer);
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2258 }
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2259
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2260 /// Scans a hexadecimal floating point number literal.
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2261 void scanHexReal(ref Token t)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2262 {
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2263 assert(*p == '.' || *p == 'p' || *p == 'P');
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2264 MID mid;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2265 if (*p == '.')
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2266 while (ishexad(*++p) || *p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2267 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2268 // Decimal exponent is required.
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2269 if (*p != 'p' && *p != 'P')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2270 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2271 mid = MID.HexFloatExponentRequired;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2272 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2273 }
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2274 // Scan exponent
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2275 assert(*p == 'p' || *p == 'P');
393
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2276 ++p;
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2277 if (*p == '+' || *p == '-')
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2278 ++p;
fce1e6133dac Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 392
diff changeset
2279 if (!isdigit(*p))
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2280 {
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2281 mid = MID.HexFloatExpMustStartWithDigit;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2282 goto Lerr;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2283 }
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2284 while (isdigit(*++p) || *p == '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2285 {}
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2286 // Copy whole number and remove underscores from buffer.
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2287 char[] buffer = t.start[0..p-t.start].dup;
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2288 uint j;
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2289 foreach (c; buffer)
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2290 if (c != '_')
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2291 buffer[j++] = c;
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2292 buffer.length = j; // Adjust length.
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2293 buffer ~= 0; // Terminate for C functions.
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2294 finalizeFloat(t, buffer);
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2295 return;
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2296 Lerr:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2297 t.kind = TOK.Float32;
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2298 t.end = p;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2299 error(t.start, mid);
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2300 }
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2301
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2302 /// Sets the value of the token.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2303 /// Params:
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2304 /// t = receives the value.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2305 /// buffer = the well-formed float number.
63
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2306 void finalizeFloat(ref Token t, string buffer)
c29229fbf2f7 - Recognizing floats that start with a dot.
aziz
parents: 62
diff changeset
2307 {
389
c4bfceab7246 Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 388
diff changeset
2308 assert(buffer[$-1] == 0);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2309 // Float number is well-formed. Check suffixes and do conversion.
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2310 switch (*p)
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2311 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2312 case 'f', 'F':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2313 t.kind = TOK.Float32;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2314 t.float_ = strtof(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2315 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2316 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2317 case 'L':
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2318 t.kind = TOK.Float80;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2319 t.real_ = strtold(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2320 ++p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2321 break;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2322 default:
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2323 t.kind = TOK.Float64;
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2324 t.double_ = strtod(buffer.ptr, null);
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2325 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2326 if (*p == 'i')
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2327 {
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2328 ++p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2329 t.kind += 3; // Switch to imaginary counterpart.
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2330 assert(t.kind == TOK.Imaginary32 ||
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2331 t.kind == TOK.Imaginary64 ||
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2332 t.kind == TOK.Imaginary80);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2333 }
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
2334 if (errno() == ERANGE)
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2335 error(t.start, MID.OverflowFloatNumber);
62
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2336 t.end = p;
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2337 }
96af5653acef - Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents: 61
diff changeset
2338
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2339 /// Scans a special token sequence.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2340 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2341 /// SpecialTokenSequence := "#line" Integer Filespec? EndOfLine
360
b6a3755eba94 - Renamed scanSpecialToken() to scanSpecialTokenSequence().
aziz
parents: 350
diff changeset
2342 void scanSpecialTokenSequence(ref Token t)
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2343 {
47
8aa37a78937b - Properly implemented scanner for normal string literals.
aziz
parents: 46
diff changeset
2344 assert(*p == '#');
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2345 t.kind = TOK.HashLine;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
2346 t.setWhitespaceFlag();
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2347
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2348 MID mid;
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2349 char* errorAtColumn = p;
763
f26f13b5a3a3 Fixed code in scanSpecialTokenSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 762
diff changeset
2350 char* tokenEnd = ++p;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2351
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2352 if (!(p[0] == 'l' && p[1] == 'i' && p[2] == 'n' && p[3] == 'e'))
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2353 {
370
ae4afb66768f - Renamed findModule() to findModulePath().
aziz
parents: 367
diff changeset
2354 mid = MID.ExpectedIdentifierSTLine;
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2355 goto Lerr;
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2356 }
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2357 p += 3;
763
f26f13b5a3a3 Fixed code in scanSpecialTokenSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 762
diff changeset
2358 tokenEnd = p + 1;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2359
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2360 // TODO: #line58"path/file" is legal. Require spaces?
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2361 // State.Space could be used for that purpose.
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2362 enum State
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2363 { /+Space,+/ Integer, Filespec, End }
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2364
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2365 State state = State.Integer;
763
f26f13b5a3a3 Fixed code in scanSpecialTokenSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 762
diff changeset
2366
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2367 while (!isEndOfLine(++p))
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2368 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2369 if (isspace(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2370 continue;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2371 if (state == State.Integer)
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2372 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2373 if (!isdigit(*p))
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2374 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2375 errorAtColumn = p;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2376 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2377 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2378 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2379 t.tokLineNum = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2380 scan(*t.tokLineNum);
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2381 tokenEnd = p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2382 if (t.tokLineNum.kind != TOK.Int32 && t.tokLineNum.kind != TOK.Uint32)
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2383 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2384 errorAtColumn = t.tokLineNum.start;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2385 mid = MID.ExpectedIntegerAfterSTLine;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2386 goto Lerr;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2387 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2388 --p; // Go one back because scan() advanced p past the integer.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2389 state = State.Filespec;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2390 }
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2391 else if (state == State.Filespec && *p == '"')
763
f26f13b5a3a3 Fixed code in scanSpecialTokenSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 762
diff changeset
2392 { // MID.ExpectedFilespec is deprecated.
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2393 // if (*p != '"')
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2394 // {
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2395 // errorAtColumn = p;
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2396 // mid = MID.ExpectedFilespec;
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2397 // goto Lerr;
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2398 // }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2399 t.tokLineFilespec = new Token;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2400 t.tokLineFilespec.start = p;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2401 t.tokLineFilespec.kind = TOK.Filespec;
552
3bc7801c207e Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 550
diff changeset
2402 t.tokLineFilespec.setWhitespaceFlag();
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2403 while (*++p != '"')
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2404 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2405 if (isEndOfLine(p))
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2406 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2407 errorAtColumn = t.tokLineFilespec.start;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2408 mid = MID.UnterminatedFilespec;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2409 t.tokLineFilespec.end = p;
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2410 tokenEnd = p;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2411 goto Lerr;
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2412 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2413 isascii(*p) || decodeUTF8();
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2414 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2415 auto start = t.tokLineFilespec.start +1; // +1 skips '"'
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2416 t.tokLineFilespec.str = start[0 .. p - start];
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2417 t.tokLineFilespec.end = p + 1;
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2418 tokenEnd = p + 1;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2419 state = State.End;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2420 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2421 else/+ if (state == State.End)+/
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2422 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2423 mid = MID.UnterminatedSpecialToken;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2424 goto Lerr;
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2425 }
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2426 }
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2427 assert(isEndOfLine(p));
323
6259fb93e3dd - Rewrote scanSpecialToken().
aziz
parents: 315
diff changeset
2428
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2429 if (state == State.Integer)
51
cadd2bfe686c - Displaying error messages in XML.
aziz
parents: 50
diff changeset
2430 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2431 errorAtColumn = p;
388
ae154eceba65 Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 387
diff changeset
2432 mid = MID.ExpectedIntegerAfterSTLine;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2433 goto Lerr;
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2434 }
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2435
392
bb935c6f9b7a Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 391
diff changeset
2436 // Evaluate #line only when not in token string.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2437 if (!inTokenString && t.tokLineNum)
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2438 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2439 this.lineNum_hline = this.lineNum - t.tokLineNum.uint_ + 1;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2440 if (t.tokLineFilespec)
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
2441 newFilePath(t.tokLineFilespec.str);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2442 }
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2443 p = tokenEnd;
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2444 t.end = tokenEnd;
30
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2445
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2446 return;
426767b94635 - Added code for parsing the '#line' special token.
aziz
parents: 29
diff changeset
2447 Lerr:
762
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2448 p = tokenEnd;
c909a3d3fa52 Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 755
diff changeset
2449 t.end = tokenEnd;
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2450 error(errorAtColumn, mid);
371
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer.
aziz
parents: 370
diff changeset
2451 }
01887f05d4b0 - Added members loc_old and loc_hline to class Lexer.
aziz
parents: 370
diff changeset
2452
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2453 /// Inserts an empty dummy token (TOK.Empty) before t.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2454 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2455 /// Useful in the parsing phase for representing a node in the AST
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2456 /// that doesn't consume an actual token from the source text.
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2457 Token* insertEmptyTokenBefore(Token* t)
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2458 {
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2459 assert(t !is null && t.prev !is null);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2460 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind));
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2461 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind));
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2462
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2463 auto prev_t = t.prev;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2464 auto new_t = new Token;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2465 new_t.kind = TOK.Empty;
414
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2466 new_t.start = new_t.end = prev_t.end;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2467 // Link in new token.
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2468 prev_t.next = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2469 new_t.prev = prev_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2470 new_t.next = t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2471 t.prev = new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2472 return new_t;
9c69615a4876 Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 413
diff changeset
2473 }
105
df34ec47fb81 - Added getState() method and State struct to Lexer.
aziz
parents: 103
diff changeset
2474
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2475 /// Returns the error line number.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2476 uint errorLineNumber(uint lineNum)
239
7911f6a92e6e - Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents: 223
diff changeset
2477 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2478 return lineNum - this.lineNum_hline;
18
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
2479 }
c48d2125f1e2 - Moved code for scanning character literals to separate function.
aziz
parents: 17
diff changeset
2480
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2481 /// Forwards error parameters.
803
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
2482 void error(char* columnPos, char[] msg, ...)
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
2483 {
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
2484 error_(this.lineNum, this.lineBegin, columnPos, msg, _arguments, _argptr);
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
2485 }
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
2486
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
2487 /// ditto
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2488 void error(char* columnPos, MID mid, ...)
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2489 {
803
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
2490 error_(this.lineNum, this.lineBegin, columnPos, GetMsg(mid), _arguments, _argptr);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2491 }
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2492
798
c24be8d4f6ab Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 797
diff changeset
2493 /// ditto
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2494 void error(uint lineNum, char* lineBegin, char* columnPos, MID mid, ...)
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2495 {
803
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
2496 error_(lineNum, lineBegin, columnPos, GetMsg(mid), _arguments, _argptr);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2497 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2498
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2499 /// Creates an error report and appends it to a list.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2500 /// Params:
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2501 /// lineNum = the line number.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2502 /// lineBegin = points to the first character of the current line.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2503 /// columnPos = points to the character where the error is located.
803
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
2504 /// msg = the message.
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
2505 void error_(uint lineNum, char* lineBegin, char* columnPos, char[] msg,
829
55c463c57d3a Fixed variable argument parameter issue.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 826
diff changeset
2506 TypeInfo[] _arguments, va_list _argptr)
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2507 {
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2508 lineNum = this.errorLineNumber(lineNum);
607
2ed1e6d638cd Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 601
diff changeset
2509 auto errorPath = this.filePaths.setPath;
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2510 auto location = new Location(errorPath, lineNum, lineBegin, columnPos);
803
cb8040538772 Reporting error for invalid octal escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 798
diff changeset
2511 msg = Format(_arguments, _argptr, msg);
520
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2512 auto error = new LexerError(location, msg);
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2513 errors ~= error;
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2514 if (infoMan !is null)
f203c5248d0b Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 519
diff changeset
2515 infoMan ~= error;
207
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2516 }
481ed2b63a49 - Added contracts to method scan().
aziz
parents: 131
diff changeset
2517
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2518 /// Scans the whole source text until EOF is encountered.
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2519 void scanAll()
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2520 {
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2521 while (nextToken() != TOK.EOF)
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2522 {}
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2523 }
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2524
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2525 /// Returns the first token of the source text.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2526 /// This can be the EOF token.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2527 /// Structure: HEAD -> Newline -> First Token
487
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2528 Token* firstToken()
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2529 {
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2530 return this.head.next.next;
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2531 }
bccca748d745 Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 485
diff changeset
2532
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2533 /// Returns true if str is a valid D identifier.
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2534 static bool isIdentifierString(char[] str)
365
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2535 {
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2536 if (str.length == 0 || isdigit(str[0]))
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2537 return false;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2538 size_t idx;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2539 do
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2540 {
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2541 auto c = dil.Unicode.decode(str, idx);
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2542 if (c == ERROR_CHAR || !(isident(c) || !isascii(c) && isUniAlpha(c)))
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2543 return false;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2544 } while (idx < str.length)
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2545 return true;
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2546 }
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2547
818
372fa4fbbb1d Added error messages and applied fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 814
diff changeset
2548 /// Returns true if str is a keyword or
372fa4fbbb1d Added error messages and applied fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 814
diff changeset
2549 /// a special token (__FILE__, __LINE__ etc.)
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2550 static bool isReservedIdentifier(char[] str)
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2551 {
818
372fa4fbbb1d Added error messages and applied fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 814
diff changeset
2552 if (str.length == 0)
372fa4fbbb1d Added error messages and applied fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 814
diff changeset
2553 return false;
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2554 auto id = IdTable.inStatic(str);
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2555 if (id is null || id.kind == TOK.Identifier)
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2556 return false; // str is not in the table or a normal identifier.
814
49e32b5bc161 Added isValidUnreservedIdentifier() to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 806
diff changeset
2557 return true;
49e32b5bc161 Added isValidUnreservedIdentifier() to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 806
diff changeset
2558 }
365
ed67acc82268 - Added option includes to config.d.
aziz
parents: 360
diff changeset
2559
814
49e32b5bc161 Added isValidUnreservedIdentifier() to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 806
diff changeset
2560 /// Returns true if this is a valid identifier and if it's not reserved.
49e32b5bc161 Added isValidUnreservedIdentifier() to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 806
diff changeset
2561 static bool isValidUnreservedIdentifier(char[] str)
49e32b5bc161 Added isValidUnreservedIdentifier() to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 806
diff changeset
2562 {
49e32b5bc161 Added isValidUnreservedIdentifier() to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 806
diff changeset
2563 return isIdentifierString(str) && !isReservedIdentifier(str);
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2564 }
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2565
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2566 /// Returns true if the current character to be decoded is
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2567 /// a Unicode alpha character.
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2568 ///
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2569 /// The current pointer 'p' is not advanced if false is returned.
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2570 bool isUnicodeAlpha()
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2571 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2572 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2573 char* p = this.p;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2574 dchar d = *p;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2575 ++p; // Move to second byte.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2576 // Error if second byte is not a trail byte.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2577 if (!isTrailByte(*p))
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2578 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2579 // Check for overlong sequences.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2580 switch (d)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2581 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2582 case 0xE0, 0xF0, 0xF8, 0xFC:
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2583 if ((*p & d) == 0x80)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2584 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2585 default:
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2586 if ((d & 0xFE) == 0xC0) // 1100000x
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2587 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2588 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2589 const char[] checkNextByte = "if (!isTrailByte(*++p))"
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2590 " return false;";
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2591 const char[] appendSixBits = "d = (d << 6) | *p & 0b0011_1111;";
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2592 // Decode
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2593 if ((d & 0b1110_0000) == 0b1100_0000)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2594 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2595 d &= 0b0001_1111;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2596 mixin(appendSixBits);
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2597 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2598 else if ((d & 0b1111_0000) == 0b1110_0000)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2599 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2600 d &= 0b0000_1111;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2601 mixin(appendSixBits ~
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2602 checkNextByte ~ appendSixBits);
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2603 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2604 else if ((d & 0b1111_1000) == 0b1111_0000)
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2605 {
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2606 d &= 0b0000_0111;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2607 mixin(appendSixBits ~
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2608 checkNextByte ~ appendSixBits ~
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2609 checkNextByte ~ appendSixBits);
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2610 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2611 else
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2612 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2613
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2614 assert(isTrailByte(*p));
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2615 if (!isValidChar(d) || !isUniAlpha(d))
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2616 return false;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2617 // Only advance pointer if this is a Unicode alpha character.
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2618 this.p = p;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2619 return true;
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2620 }
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2621
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2622 /// Decodes the next UTF-8 sequence.
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2623 dchar decodeUTF8()
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2624 {
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2625 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8().");
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2626 char* p = this.p;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2627 dchar d = *p;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2628
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2629 ++p; // Move to second byte.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2630 // Error if second byte is not a trail byte.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2631 if (!isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2632 goto Lerr2;
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2633
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2634 // Check for overlong sequences.
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2635 switch (d)
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2636 {
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2637 case 0xE0, // 11100000 100xxxxx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2638 0xF0, // 11110000 1000xxxx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2639 0xF8, // 11111000 10000xxx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2640 0xFC: // 11111100 100000xx
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2641 if ((*p & d) == 0x80)
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2642 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2643 default:
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2644 if ((d & 0xFE) == 0xC0) // 1100000x
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2645 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2646 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2647
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2648 const char[] checkNextByte = "if (!isTrailByte(*++p))"
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2649 " goto Lerr2;";
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2650 const char[] appendSixBits = "d = (d << 6) | *p & 0b0011_1111;";
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2651
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2652 // Decode
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2653 if ((d & 0b1110_0000) == 0b1100_0000)
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2654 { // 110xxxxx 10xxxxxx
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2655 d &= 0b0001_1111;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2656 mixin(appendSixBits);
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2657 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2658 else if ((d & 0b1111_0000) == 0b1110_0000)
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2659 { // 1110xxxx 10xxxxxx 10xxxxxx
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2660 d &= 0b0000_1111;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2661 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2662 checkNextByte ~ appendSixBits);
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2663 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2664 else if ((d & 0b1111_1000) == 0b1111_0000)
550
082126d78f90 Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 538
diff changeset
2665 { // 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2666 d &= 0b0000_0111;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2667 mixin(appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2668 checkNextByte ~ appendSixBits ~
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2669 checkNextByte ~ appendSixBits);
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2670 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2671 else
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2672 // 5 and 6 byte UTF-8 sequences are not allowed yet.
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2673 // 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2674 // 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2675 goto Lerr;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2676
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2677 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2678
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2679 if (!isValidChar(d))
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2680 {
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2681 Lerr:
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2682 // Three cases:
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2683 // *) the UTF-8 sequence was successfully decoded but the resulting
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2684 // character is invalid.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2685 // p points to last trail byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2686 // *) the UTF-8 sequence is overlong.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2687 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2688 // *) the UTF-8 sequence has more than 4 bytes or starts with
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2689 // a trail byte.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2690 // p points to second byte in the sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2691 assert(isTrailByte(*p));
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2692 // Move to next ASCII character or lead byte of a UTF-8 sequence.
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2693 while (p < (end-1) && isTrailByte(*p))
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2694 ++p;
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2695 --p;
426
3f7790d3f9d6 Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 425
diff changeset
2696 assert(!isTrailByte(p[1]));
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2697 Lerr2:
430
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 426
diff changeset
2698 d = REPLACEMENT_CHAR;
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2699 error(this.p, MID.InvalidUTF8Sequence, formatBytes(this.p, p));
425
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2700 }
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2701
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2702 this.p = p;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2703 return d;
6bf936bf3356 Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 424
diff changeset
2704 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2705
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2706 /// Encodes the character d and appends it to str.
519
9ebc799c7dc5 Fixes in dil.Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 514
diff changeset
2707 static void encodeUTF8(ref char[] str, dchar d)
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2708 {
430
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 426
diff changeset
2709 assert(!isascii(d), "check for ASCII char before calling encodeUTF8().");
510
dd3ce87b3569 Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 507
diff changeset
2710 assert(isValidChar(d), "check if character is valid before calling encodeUTF8().");
430
e6c759e151cd Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 426
diff changeset
2711
519
9ebc799c7dc5 Fixes in dil.Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 514
diff changeset
2712 char[6] b = void;
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2713 if (d < 0x800)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2714 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2715 b[0] = 0xC0 | (d >> 6);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2716 b[1] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2717 str ~= b[0..2];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2718 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2719 else if (d < 0x10000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2720 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2721 b[0] = 0xE0 | (d >> 12);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2722 b[1] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2723 b[2] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2724 str ~= b[0..3];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2725 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2726 else if (d < 0x200000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2727 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2728 b[0] = 0xF0 | (d >> 18);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2729 b[1] = 0x80 | ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2730 b[2] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2731 b[3] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2732 str ~= b[0..4];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2733 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2734 /+ // There are no 5 and 6 byte UTF-8 sequences yet.
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2735 else if (d < 0x4000000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2736 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2737 b[0] = 0xF8 | (d >> 24);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2738 b[1] = 0x80 | ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2739 b[2] = 0x80 | ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2740 b[3] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2741 b[4] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2742 str ~= b[0..5];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2743 }
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2744 else if (d < 0x80000000)
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2745 {
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2746 b[0] = 0xFC | (d >> 30);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2747 b[1] = 0x80 | ((d >> 24) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2748 b[2] = 0x80 | ((d >> 18) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2749 b[3] = 0x80 | ((d >> 12) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2750 b[4] = 0x80 | ((d >> 6) & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2751 b[5] = 0x80 | (d & 0x3F);
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2752 str ~= b[0..6];
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2753 }
424
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2754 +/
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2755 else
bb3cb00feeb2 Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 422
diff changeset
2756 assert(0);
48
c2e0e0269c28 - Added code for scanning escape string literals.
aziz
parents: 47
diff changeset
2757 }
789
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2758
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2759 /// Formats the bytes between start and end.
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2760 /// Returns: e.g.: abc -> \x61\x62\x63
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2761 static char[] formatBytes(char* start, char* end)
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2762 {
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2763 auto strLen = end-start;
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2764 const formatLen = `\xXX`.length;
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2765 char[] result = new char[strLen*formatLen]; // Reserve space.
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2766 result.length = 0;
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2767 foreach (c; cast(ubyte[])start[0..strLen])
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2768 result ~= Format("\\x{:X}", c);
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2769 return result;
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2770 }
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2771
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2772 /// Searches for an invalid UTF-8 sequence in str.
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2773 /// Returns: a formatted string of the invalid sequence (e.g. \xC0\x80).
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2774 static string findInvalidUTF8Sequence(string str)
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2775 {
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2776 char* p = str.ptr, end = p + str.length;
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2777 while (p < end)
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2778 {
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2779 if (decode(p, end) == ERROR_CHAR)
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2780 {
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2781 auto begin = p;
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2782 // Skip trail-bytes.
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2783 while (++p < end && isTrailByte(*p))
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2784 {}
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2785 return Lexer.formatBytes(begin, p);
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2786 }
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2787 }
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2788 assert(p == end);
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2789 return "";
c1d5cfd7aa44 Implemented string literal conversion.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 776
diff changeset
2790 }
5
79b4e8848794 - Started writing XML generator.
aziz
parents: 4
diff changeset
2791 }
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2792
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2793 /// Tests the lexer with a list of tokens.
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2794 unittest
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2795 {
391
33b566df6af4 Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 390
diff changeset
2796 Stdout("Testing Lexer.\n");
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2797 struct Pair
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2798 {
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2799 char[] tokenText;
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2800 TOK kind;
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2801 }
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2802 static Pair[] pairs = [
494
9a7ca8c56e59 Refactored a few things in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
2803 {"#!äöüß", TOK.Shebang}, {"\n", TOK.Newline},
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2804 {"//çay", TOK.Comment}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2805 {"&", TOK.AndBinary},
413
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2806 {"/*çağ*/", TOK.Comment}, {"&&", TOK.AndLogical},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2807 {"/+çak+/", TOK.Comment}, {"&=", TOK.AndAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2808 {">", TOK.Greater}, {"+", TOK.Plus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2809 {">=", TOK.GreaterEqual}, {"++", TOK.PlusPlus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2810 {">>", TOK.RShift}, {"+=", TOK.PlusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2811 {">>=", TOK.RShiftAssign}, {"-", TOK.Minus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2812 {">>>", TOK.URShift}, {"--", TOK.MinusMinus},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2813 {">>>=", TOK.URShiftAssign}, {"-=", TOK.MinusAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2814 {"<", TOK.Less}, {"=", TOK.Assign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2815 {"<=", TOK.LessEqual}, {"==", TOK.Equal},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2816 {"<>", TOK.LorG}, {"~", TOK.Tilde},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2817 {"<>=", TOK.LorEorG}, {"~=", TOK.CatAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2818 {"<<", TOK.LShift}, {"*", TOK.Mul},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2819 {"<<=", TOK.LShiftAssign}, {"*=", TOK.MulAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2820 {"!", TOK.Not}, {"/", TOK.Div},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2821 {"!=", TOK.NotEqual}, {"/=", TOK.DivAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2822 {"!<", TOK.UorGorE}, {"^", TOK.Xor},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2823 {"!>", TOK.UorLorE}, {"^=", TOK.XorAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2824 {"!<=", TOK.UorG}, {"%", TOK.Mod},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2825 {"!>=", TOK.UorL}, {"%=", TOK.ModAssign},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2826 {"!<>", TOK.UorE}, {"(", TOK.LParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2827 {"!<>=", TOK.Unordered}, {")", TOK.RParen},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2828 {".", TOK.Dot}, {"[", TOK.LBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2829 {"..", TOK.Slice}, {"]", TOK.RBracket},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2830 {"...", TOK.Ellipses}, {"{", TOK.LBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2831 {"|", TOK.OrBinary}, {"}", TOK.RBrace},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2832 {"||", TOK.OrLogical}, {":", TOK.Colon},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2833 {"|=", TOK.OrAssign}, {";", TOK.Semicolon},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2834 {"?", TOK.Question}, {",", TOK.Comma},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2835 {"$", TOK.Dollar}, {"cam", TOK.Identifier},
0fd78fdcb982 Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 412
diff changeset
2836 {"çay", TOK.Identifier}, {".0", TOK.Float64},
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2837 {"0", TOK.Int32}, {"\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2838 {"\r", TOK.Newline}, {"\r\n", TOK.Newline},
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2839 {"\u2028", TOK.Newline}, {"\u2029", TOK.Newline}
40
9d5ceb0f8be9 - Added more tokens for testing.
aziz
parents: 39
diff changeset
2840 ];
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2841
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2842 char[] src;
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2843
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2844 // Join all token texts into a single string.
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2845 foreach (i, pair; pairs)
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2846 if (pair.kind == TOK.Comment && pair.tokenText[1] == '/' || // Line comment.
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2847 pair.kind == TOK.Shebang)
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2848 {
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2849 assert(pairs[i+1].kind == TOK.Newline); // Must be followed by a newline.
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2850 src ~= pair.tokenText;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2851 }
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2852 else
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2853 src ~= pair.tokenText ~ " ";
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2854
826
d659f7aa055c Fixed Lexer unittest.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 818
diff changeset
2855 // Lex the constructed source text.
764
4579e8505d5e Fixed unittests and removed dil.File.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 763
diff changeset
2856 auto lx = new Lexer(new SourceText("", src));
826
d659f7aa055c Fixed Lexer unittest.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 818
diff changeset
2857 lx.scanAll();
d659f7aa055c Fixed Lexer unittest.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 818
diff changeset
2858
d659f7aa055c Fixed Lexer unittest.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 818
diff changeset
2859 auto token = lx.firstToken();
39
69b940398d7b - Added unittest to test correct parsing of operator tokens.
aziz
parents: 38
diff changeset
2860
826
d659f7aa055c Fixed Lexer unittest.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 818
diff changeset
2861 for (uint i; i < pairs.length && token.kind != TOK.EOF;
d659f7aa055c Fixed Lexer unittest.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 818
diff changeset
2862 ++i, (token = token.next))
d659f7aa055c Fixed Lexer unittest.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 818
diff changeset
2863 if (token.srcText != pairs[i].tokenText)
d659f7aa055c Fixed Lexer unittest.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 818
diff changeset
2864 assert(0, Format("Scanned '{0}' but expected '{1}'",
d659f7aa055c Fixed Lexer unittest.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 818
diff changeset
2865 token.srcText, pairs[i].tokenText));
41
2b7be1d67d4d - Optimized scanner of block comments.
aziz
parents: 40
diff changeset
2866 }
55
5887751f8e04 - Relocated ptable to the bottom of the source file.
aziz
parents: 54
diff changeset
2867
797
cf2ad5df025c Added documentation comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 789
diff changeset
2868 /// Tests the Lexer's peek() method.
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2869 unittest
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2870 {
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2871 Stdout("Testing method Lexer.peek()\n");
764
4579e8505d5e Fixed unittests and removed dil.File.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 763
diff changeset
2872 auto sourceText = new SourceText("", "unittest { }");
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2873 auto lx = new Lexer(sourceText, null);
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2874
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2875 auto next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2876 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2877 assert(next.kind == TOK.Newline);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2878 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2879 assert(next.kind == TOK.Unittest);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2880 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2881 assert(next.kind == TOK.LBrace);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2882 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2883 assert(next.kind == TOK.RBrace);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2884 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2885 assert(next.kind == TOK.EOF);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2886
764
4579e8505d5e Fixed unittests and removed dil.File.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 763
diff changeset
2887 lx = new Lexer(new SourceText("", ""));
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2888 next = lx.head;
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2889 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2890 assert(next.kind == TOK.Newline);
485
ea8c7459f1c4 Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 464
diff changeset
2891 lx.peek(next);
679
ff6971637f88 Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 607
diff changeset
2892 assert(next.kind == TOK.EOF);
422
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2893 }
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2894
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2895 unittest
ad7977fe315a Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 420
diff changeset
2896 {
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2897 // Numbers unittest
68
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2898 // 0L 0ULi 0_L 0_UL 0x0U 0x0p2 0_Fi 0_e2 0_F 0_i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2899 // 0u 0U 0uL 0UL 0L 0LU 0Lu
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2900 // 0Li 0f 0F 0fi 0Fi 0i
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2901 // 0b_1_LU 0b1000u
7eb83dd38901 - Simplified suffix rule and added a few more numbers to unittest.
aziz
parents: 67
diff changeset
2902 // 0x232Lu
67
996065105910 - Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents: 66
diff changeset
2903 }