Mercurial > projects > dil
annotate trunk/src/dil/Lexer.d @ 532:50e64bab9c7a
Renamed InformationManager to InfoManager.
author | Aziz K?ksal <aziz.koeksal@gmail.com> |
---|---|
date | Mon, 17 Dec 2007 16:10:08 +0100 |
parents | f203c5248d0b |
children | d0d40bcca9c6 |
rev | line source |
---|---|
0 | 1 /++ |
2 Author: Aziz Köksal | |
249 | 3 License: GPL3 |
0 | 4 +/ |
326 | 5 module dil.Lexer; |
327 | 6 import dil.Token; |
7 import dil.Information; | |
8 import dil.Keywords; | |
9 import dil.Identifier; | |
10 import dil.Messages; | |
11 import dil.HtmlEntities; | |
464
325714d8aa6c
Added new module with compiler constants to resolve import cycle.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
434
diff
changeset
|
12 import dil.CompilerInfo; |
502
4e14cd1b24da
Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
499
diff
changeset
|
13 import dil.IdTable; |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
14 import dil.Unicode; |
391
33b566df6af4
Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
390
diff
changeset
|
15 import tango.stdc.stdlib : strtof, strtod, strtold; |
33b566df6af4
Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
390
diff
changeset
|
16 import tango.stdc.errno : errno, ERANGE; |
33b566df6af4
Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
390
diff
changeset
|
17 import tango.stdc.time : time_t, time, ctime; |
33b566df6af4
Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
390
diff
changeset
|
18 import tango.stdc.string : strlen; |
33b566df6af4
Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
390
diff
changeset
|
19 import common; |
0 | 20 |
490
47be6bfe39cd
Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
489
diff
changeset
|
21 public import dil.LexerFuncs; |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
22 |
499
52447db67938
Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
498
diff
changeset
|
23 /++ |
52447db67938
Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
498
diff
changeset
|
24 The Lexer analyzes the characters of a source text and |
52447db67938
Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
498
diff
changeset
|
25 produces a doubly-linked list of tokens. |
52447db67938
Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
498
diff
changeset
|
26 +/ |
0 | 27 class Lexer |
28 { | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
29 Token* head; /// The head of the doubly linked token list. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
30 Token* tail; /// The tail of the linked list. Set in scan(). |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
31 Token* token; /// Points to the current token in the token list. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
32 string text; /// The source text. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
33 char[] filePath; /// Path to the source text. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
34 char* p; /// Points to the current character in the source text. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
35 char* end; /// Points one character past the end of the source text. |
4 | 36 |
434
cac9c8d637ad
Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
433
diff
changeset
|
37 // Members used for error messages: |
532
50e64bab9c7a
Renamed InformationManager to InfoManager.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
520
diff
changeset
|
38 InfoManager infoMan; |
514
6ddff941862a
Added new error classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
510
diff
changeset
|
39 LexerError[] errors; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
40 /// Always points to the beginning of the current line. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
41 char* lineBegin; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
42 // Token* newline; /// Current newline token. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
43 uint lineNum = 1; /// Current, actual source text line number. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
44 uint lineNum_hline; /// Line number set by #line. |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
45 uint inTokenString; /// > 0 if inside q{ } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
46 char[] errorPath; /// The path displayed in error messages. |
103
511a1aa25896
- Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too.
aziz
parents:
94
diff
changeset
|
47 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
48 /++ |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
49 Construct a Lexer object. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
50 Params: |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
51 text = the UTF-8 source code. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
52 filePath = the path to the source code; used for error messages. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
53 +/ |
532
50e64bab9c7a
Renamed InformationManager to InfoManager.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
520
diff
changeset
|
54 this(string text, string filePath, InfoManager infoMan = null) |
4 | 55 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
56 this.filePath = this.errorPath = filePath; |
520
f203c5248d0b
Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
519
diff
changeset
|
57 this.infoMan = infoMan; |
30 | 58 |
4 | 59 this.text = text; |
397
c99f8aeb7b4a
Empty source files are handled correctly now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
396
diff
changeset
|
60 if (text.length == 0 || text[$-1] != 0) |
39
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
61 { |
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
62 this.text.length = this.text.length + 1; |
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
63 this.text[$-1] = 0; |
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
64 } |
4 | 65 |
66 this.p = this.text.ptr; | |
67 this.end = this.p + this.text.length; | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
68 this.lineBegin = this.p; |
239
7911f6a92e6e
- Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents:
223
diff
changeset
|
69 |
7911f6a92e6e
- Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents:
223
diff
changeset
|
70 this.head = new Token; |
7911f6a92e6e
- Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents:
223
diff
changeset
|
71 this.head.type = TOK.HEAD; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
72 this.head.start = this.head.end = this.p; |
239
7911f6a92e6e
- Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents:
223
diff
changeset
|
73 this.token = this.head; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
74 // Add a newline as the first token after the head. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
75 auto newline = new Token; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
76 newline.type = TOK.Newline; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
77 newline.start = newline.end = this.p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
78 newline.filePath = this.errorPath; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
79 newline.lineNum = 1; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
80 newline.lineNum_hline = 0; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
81 // Link in. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
82 this.token.next = newline; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
83 newline.prev = this.token; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
84 this.token = newline; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
85 // this.newline = newline; |
315 | 86 scanShebang(); |
87 } | |
88 | |
344 | 89 ~this() |
90 { | |
91 auto token = head.next; | |
418
7354f15cd5e9
Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
416
diff
changeset
|
92 while (token !is null) |
344 | 93 { |
94 assert(token.type == TOK.EOF ? token == tail && token.next is null : 1); | |
95 delete token.prev; | |
96 token = token.next; | |
418
7354f15cd5e9
Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
416
diff
changeset
|
97 } |
344 | 98 delete tail; |
99 } | |
100 | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
101 /++ |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
102 The "shebang" may optionally appear once at the beginning of a file. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
103 Regexp: #![^\EndOfLine]* |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
104 +/ |
315 | 105 void scanShebang() |
106 { | |
107 if (*p == '#' && p[1] == '!') | |
108 { | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
109 auto t = new Token; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
110 t.type = TOK.Shebang; |
315 | 111 t.start = p; |
112 ++p; | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
113 while (!isEndOfLine(++p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
114 isascii(*p) || decodeUTF8(); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
115 t.end = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
116 this.token.next = t; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
117 t.prev = this.token; |
315 | 118 } |
4 | 119 } |
120 | |
343
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
121 void finalizeSpecialToken(ref Token t) |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
122 { |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
123 assert(t.srcText[0..2] == "__"); |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
124 switch (t.type) |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
125 { |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
126 case TOK.FILE: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
127 t.str = this.errorPath; |
343
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
128 break; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
129 case TOK.LINE: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
130 t.uint_ = this.errorLineNumber(this.lineNum); |
343
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
131 break; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
132 case TOK.DATE, |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
133 TOK.TIME, |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
134 TOK.TIMESTAMP: |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
135 time_t time_val; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
136 time(&time_val); |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
137 char* str = ctime(&time_val); |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
138 char[] time_str = str[0 .. strlen(str)]; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
139 switch (t.type) |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
140 { |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
141 case TOK.DATE: |
346
ce9e2e77743f
- Fix: include one space when slicing time_str for __DATE__.
aziz
parents:
344
diff
changeset
|
142 time_str = time_str[4..11] ~ time_str[20..24] ~ \0; break; |
343
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
143 case TOK.TIME: |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
144 time_str = time_str[11..19] ~ \0; break; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
145 case TOK.TIMESTAMP: |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
146 time_str = time_str[0..24] ~ \0; break; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
147 default: assert(0); |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
148 } |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
149 t.str = time_str; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
150 break; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
151 case TOK.VENDOR: |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
152 t.str = VENDOR; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
153 break; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
154 case TOK.VERSION: |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
155 t.uint_ = VERSION_MAJOR*1000 + VERSION_MINOR; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
156 break; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
157 default: |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
158 assert(0); |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
159 } |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
160 } |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
161 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
162 private void setLineBegin(char* p) |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
163 { |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
164 // Check that we can look behind one character. |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
165 assert((p-1) >= text.ptr && p < end); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
166 // Check that previous character is a newline. |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
167 assert(isNewlineEnd(p - 1)); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
168 this.lineBegin = p; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
169 } |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
170 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
171 private void scanNext(ref Token* t) |
419
89e40d43065d
Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
418
diff
changeset
|
172 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
173 assert(t !is null); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
174 if (t.next) |
434
cac9c8d637ad
Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
433
diff
changeset
|
175 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
176 t = t.next; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
177 // if (t.type == TOK.Newline) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
178 // this.newline = t; |
434
cac9c8d637ad
Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
433
diff
changeset
|
179 } |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
180 else if (t != this.tail) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
181 { |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
182 Token* new_t = new Token; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
183 scan(*new_t); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
184 new_t.prev = t; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
185 t.next = new_t; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
186 t = new_t; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
187 } |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
188 } |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
189 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
190 /// Advance t one token forward. |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
191 void peek(ref Token* t) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
192 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
193 scanNext(t); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
194 } |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
195 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
196 /// Advance to the next token in the source text. |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
197 TOK nextToken() |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
198 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
199 scanNext(this.token); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
200 return this.token.type; |
419
89e40d43065d
Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
418
diff
changeset
|
201 } |
89e40d43065d
Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
418
diff
changeset
|
202 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
203 /// Returns true if p points to the last character of a Newline. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
204 bool isNewlineEnd(char* p) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
205 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
206 if (*p == '\n' || *p == '\r') |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
207 return true; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
208 if (*p == LS[2] || *p == PS[2]) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
209 if ((p-2) >= text.ptr) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
210 if (p[-1] == LS[1] && p[-2] == LS[0]) |
434
cac9c8d637ad
Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
433
diff
changeset
|
211 return true; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
212 return false; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
213 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
214 |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
215 /++ |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
216 This is the old scan method. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
217 TODO: profile old and new to see which one is faster. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
218 +/ |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
219 public void scan(ref Token t) |
207 | 220 in |
3 | 221 { |
207 | 222 assert(text.ptr <= p && p < end); |
223 } | |
224 out | |
225 { | |
397
c99f8aeb7b4a
Empty source files are handled correctly now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
396
diff
changeset
|
226 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type)); |
c99f8aeb7b4a
Empty source files are handled correctly now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
396
diff
changeset
|
227 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type)); |
207 | 228 } |
229 body | |
230 { | |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
231 // Scan whitespace. |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
232 if (isspace(*p)) |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
233 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
234 t.ws = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
235 while (isspace(*++p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
236 {} |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
237 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
238 |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
239 // Scan a token. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
240 uint c = *p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
241 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
242 t.start = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
243 // Newline. |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
244 switch (*p) |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
245 { |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
246 case '\r': |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
247 if (p[1] == '\n') |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
248 ++p; |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
249 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
250 assert(isNewlineEnd(p)); |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
251 ++p; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
252 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
253 setLineBegin(p); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
254 // this.newline = &t; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
255 t.type = TOK.Newline; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
256 t.filePath = this.errorPath; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
257 t.lineNum = lineNum; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
258 t.lineNum_hline = lineNum_hline; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
259 t.end = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
260 return; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
261 default: |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
262 if (isUnicodeNewline(p)) |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
263 { |
412
fb31af0fda73
Added struct Location, and token2LocTable to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
411
diff
changeset
|
264 ++p; ++p; |
fb31af0fda73
Added struct Location, and token2LocTable to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
411
diff
changeset
|
265 goto case '\n'; |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
266 } |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
267 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
268 // Identifier or string literal. |
13 | 269 if (isidbeg(c)) |
5 | 270 { |
33 | 271 if (c == 'r' && p[1] == '"' && ++p) |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
272 return scanRawStringLiteral(t); |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
273 if (c == 'x' && p[1] == '"') |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
274 return scanHexStringLiteral(t); |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
275 version(D2) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
276 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
277 if (c == 'q' && p[1] == '"') |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
278 return scanDelimitedStringLiteral(t); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
279 if (c == 'q' && p[1] == '{') |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
280 return scanTokenStringLiteral(t); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
281 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
282 // Scan identifier. |
12
0989206cf73c
- Added code to decode Unicode characters in identifiers.
aziz
parents:
11
diff
changeset
|
283 Lidentifier: |
5 | 284 do |
285 { c = *++p; } | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
286 while (isident(c) || !isascii(c) && isUniAlpha(decodeUTF8())) |
28 | 287 |
5 | 288 t.end = p; |
28 | 289 |
502
4e14cd1b24da
Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
499
diff
changeset
|
290 auto id = IdTable.lookup(t.srcText); |
499
52447db67938
Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
498
diff
changeset
|
291 t.type = id.type; |
502
4e14cd1b24da
Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
499
diff
changeset
|
292 t.ident = id; |
28 | 293 |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
294 if (t.type == TOK.Identifier || t.isKeyword) |
411 | 295 return; |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
296 else if (t.isSpecialToken) |
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
297 finalizeSpecialToken(t); |
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
298 else if (t.type == TOK.EOF) |
411 | 299 { |
300 tail = &t; | |
301 assert(t.srcText == "__EOF__"); | |
302 } | |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
303 else |
502
4e14cd1b24da
Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
499
diff
changeset
|
304 assert(0, "unexpected token type: " ~ Token.toString(t.type)); |
5 | 305 return; |
306 } | |
307 | |
15 | 308 if (isdigit(c)) |
309 return scanNumber(t); | |
310 | |
8 | 311 if (c == '/') |
5 | 312 { |
8 | 313 c = *++p; |
14 | 314 switch(c) |
5 | 315 { |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
316 case '=': |
14 | 317 ++p; |
32
d7011daa4740
- Added missing commas after the items in the messages table.
aziz
parents:
31
diff
changeset
|
318 t.type = TOK.DivAssign; |
14 | 319 t.end = p; |
320 return; | |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
321 case '+': |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
322 return scanNestedComment(t); |
32
d7011daa4740
- Added missing commas after the items in the messages table.
aziz
parents:
31
diff
changeset
|
323 case '*': |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
324 return scanBlockComment(t); |
32
d7011daa4740
- Added missing commas after the items in the messages table.
aziz
parents:
31
diff
changeset
|
325 case '/': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
326 while (!isEndOfLine(++p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
327 isascii(*p) || decodeUTF8(); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
328 t.type = TOK.Comment; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
329 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
330 return; |
32
d7011daa4740
- Added missing commas after the items in the messages table.
aziz
parents:
31
diff
changeset
|
331 default: |
d7011daa4740
- Added missing commas after the items in the messages table.
aziz
parents:
31
diff
changeset
|
332 t.type = TOK.Div; |
10 | 333 t.end = p; |
334 return; | |
335 } | |
5 | 336 } |
337 | |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
338 switch (c) |
20 | 339 { |
48 | 340 case '\'': |
341 return scanCharacterLiteral(t); | |
342 case '`': | |
343 return scanRawStringLiteral(t); | |
344 case '"': | |
345 return scanNormalStringLiteral(t); | |
346 case '\\': | |
347 char[] buffer; | |
348 do | |
349 { | |
350 c = scanEscapeSequence(); | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
351 if (isascii(c)) |
48 | 352 buffer ~= c; |
353 else | |
354 encodeUTF8(buffer, c); | |
355 } while (*p == '\\') | |
356 buffer ~= 0; | |
357 t.type = TOK.String; | |
358 t.str = buffer; | |
359 t.end = p; | |
360 return; | |
38
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
361 case '>': /* > >= >> >>= >>> >>>= */ |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
362 c = *++p; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
363 switch (c) |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
364 { |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
365 case '=': |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
366 t.type = TOK.GreaterEqual; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
367 goto Lcommon; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
368 case '>': |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
369 if (p[1] == '>') |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
370 { |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
371 ++p; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
372 if (p[1] == '=') |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
373 { ++p; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
374 t.type = TOK.URShiftAssign; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
375 } |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
376 else |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
377 t.type = TOK.URShift; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
378 } |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
379 else if (p[1] == '=') |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
380 { |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
381 ++p; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
382 t.type = TOK.RShiftAssign; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
383 } |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
384 else |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
385 t.type = TOK.RShift; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
386 goto Lcommon; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
387 default: |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
388 t.type = TOK.Greater; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
389 goto Lcommon2; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
390 } |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
391 assert(0); |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
392 case '<': /* < <= <> <>= << <<= */ |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
393 c = *++p; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
394 switch (c) |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
395 { |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
396 case '=': |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
397 t.type = TOK.LessEqual; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
398 goto Lcommon; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
399 case '<': |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
400 if (p[1] == '=') { |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
401 ++p; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
402 t.type = TOK.LShiftAssign; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
403 } |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
404 else |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
405 t.type = TOK.LShift; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
406 goto Lcommon; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
407 case '>': |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
408 if (p[1] == '=') { |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
409 ++p; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
410 t.type = TOK.LorEorG; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
411 } |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
412 else |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
413 t.type = TOK.LorG; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
414 goto Lcommon; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
415 default: |
38
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
416 t.type = TOK.Less; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
417 goto Lcommon2; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
418 } |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
419 assert(0); |
37
7f3bcb97d017
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
36
diff
changeset
|
420 case '!': /* ! !< !> !<= !>= !<> !<>= */ |
35
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
421 c = *++p; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
422 switch (c) |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
423 { |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
424 case '<': |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
425 c = *++p; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
426 if (c == '>') |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
427 { |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
428 if (p[1] == '=') { |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
429 ++p; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
430 t.type = TOK.Unordered; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
431 } |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
432 else |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
433 t.type = TOK.UorE; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
434 } |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
435 else if (c == '=') |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
436 { |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
437 t.type = TOK.UorG; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
438 } |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
439 else { |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
440 t.type = TOK.UorGorE; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
441 goto Lcommon2; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
442 } |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
443 goto Lcommon; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
444 case '>': |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
445 if (p[1] == '=') |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
446 { |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
447 ++p; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
448 t.type = TOK.UorL; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
449 } |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
450 else |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
451 t.type = TOK.UorLorE; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
452 goto Lcommon; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
453 case '=': |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
454 t.type = TOK.NotEqual; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
455 goto Lcommon; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
456 default: |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
457 t.type = TOK.Not; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
458 goto Lcommon2; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
459 } |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
460 assert(0); |
63 | 461 case '.': /* . .[0-9] .. ... */ |
22
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
462 if (p[1] == '.') |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
463 { |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
464 ++p; |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
465 if (p[1] == '.') { |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
466 ++p; |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
467 t.type = TOK.Ellipses; |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
468 } |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
469 else |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
470 t.type = TOK.Slice; |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
471 } |
63 | 472 else if (isdigit(p[1])) |
473 { | |
474 return scanReal(t); | |
475 } | |
22
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
476 else |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
477 t.type = TOK.Dot; |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
478 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
479 case '|': /* | || |= */ |
23
1a7903701a3d
- Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents:
22
diff
changeset
|
480 c = *++p; |
1a7903701a3d
- Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents:
22
diff
changeset
|
481 if (c == '=') |
1a7903701a3d
- Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents:
22
diff
changeset
|
482 t.type = TOK.OrAssign; |
1a7903701a3d
- Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents:
22
diff
changeset
|
483 else if (c == '|') |
1a7903701a3d
- Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents:
22
diff
changeset
|
484 t.type = TOK.OrLogical; |
26 | 485 else { |
23
1a7903701a3d
- Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents:
22
diff
changeset
|
486 t.type = TOK.OrBinary; |
26 | 487 goto Lcommon2; |
488 } | |
23
1a7903701a3d
- Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents:
22
diff
changeset
|
489 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
490 case '&': /* & && &= */ |
24
903f91163f23
- Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents:
23
diff
changeset
|
491 c = *++p; |
903f91163f23
- Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents:
23
diff
changeset
|
492 if (c == '=') |
903f91163f23
- Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents:
23
diff
changeset
|
493 t.type = TOK.AndAssign; |
903f91163f23
- Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents:
23
diff
changeset
|
494 else if (c == '&') |
903f91163f23
- Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents:
23
diff
changeset
|
495 t.type = TOK.AndLogical; |
26 | 496 else { |
24
903f91163f23
- Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents:
23
diff
changeset
|
497 t.type = TOK.AndBinary; |
26 | 498 goto Lcommon2; |
499 } | |
24
903f91163f23
- Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents:
23
diff
changeset
|
500 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
501 case '+': /* + ++ += */ |
25
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
502 c = *++p; |
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
503 if (c == '=') |
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
504 t.type = TOK.PlusAssign; |
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
505 else if (c == '+') |
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
506 t.type = TOK.PlusPlus; |
26 | 507 else { |
25
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
508 t.type = TOK.Plus; |
26 | 509 goto Lcommon2; |
510 } | |
25
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
511 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
512 case '-': /* - -- -= */ |
25
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
513 c = *++p; |
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
514 if (c == '=') |
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
515 t.type = TOK.MinusAssign; |
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
516 else if (c == '-') |
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
517 t.type = TOK.MinusMinus; |
26 | 518 else { |
519 t.type = TOK.Minus; | |
520 goto Lcommon2; | |
521 } | |
522 goto Lcommon; | |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
523 case '=': /* = == */ |
26 | 524 if (p[1] == '=') { |
525 ++p; | |
526 t.type = TOK.Equal; | |
527 } | |
25
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
528 else |
26 | 529 t.type = TOK.Assign; |
25
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
530 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
531 case '~': /* ~ ~= */ |
27 | 532 if (p[1] == '=') { |
533 ++p; | |
534 t.type = TOK.CatAssign; | |
535 } | |
536 else | |
537 t.type = TOK.Tilde; | |
538 goto Lcommon; | |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
539 case '*': /* * *= */ |
29
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
540 if (p[1] == '=') { |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
541 ++p; |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
542 t.type = TOK.MulAssign; |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
543 } |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
544 else |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
545 t.type = TOK.Mul; |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
546 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
547 case '^': /* ^ ^= */ |
29
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
548 if (p[1] == '=') { |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
549 ++p; |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
550 t.type = TOK.XorAssign; |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
551 } |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
552 else |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
553 t.type = TOK.Xor; |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
554 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
555 case '%': /* % %= */ |
29
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
556 if (p[1] == '=') { |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
557 ++p; |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
558 t.type = TOK.ModAssign; |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
559 } |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
560 else |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
561 t.type = TOK.Mod; |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
562 goto Lcommon; |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
563 // Single character tokens: |
20 | 564 case '(': |
565 t.type = TOK.LParen; | |
566 goto Lcommon; | |
567 case ')': | |
568 t.type = TOK.RParen; | |
569 goto Lcommon; | |
570 case '[': | |
571 t.type = TOK.LBracket; | |
572 goto Lcommon; | |
573 case ']': | |
574 t.type = TOK.RBracket; | |
575 goto Lcommon; | |
576 case '{': | |
577 t.type = TOK.LBrace; | |
578 goto Lcommon; | |
579 case '}': | |
580 t.type = TOK.RBrace; | |
21
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
581 goto Lcommon; |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
582 case ':': |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
583 t.type = TOK.Colon; |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
584 goto Lcommon; |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
585 case ';': |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
586 t.type = TOK.Semicolon; |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
587 goto Lcommon; |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
588 case '?': |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
589 t.type = TOK.Question; |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
590 goto Lcommon; |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
591 case ',': |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
592 t.type = TOK.Comma; |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
593 goto Lcommon; |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
594 case '$': |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
595 t.type = TOK.Dollar; |
20 | 596 Lcommon: |
597 ++p; | |
26 | 598 Lcommon2: |
20 | 599 t.end = p; |
600 return; | |
30 | 601 case '#': |
360
b6a3755eba94
- Renamed scanSpecialToken() to scanSpecialTokenSequence().
aziz
parents:
350
diff
changeset
|
602 return scanSpecialTokenSequence(t); |
20 | 603 default: |
604 } | |
605 | |
411 | 606 // Check for EOF |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
607 if (isEOF(c)) |
411 | 608 { |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
609 assert(isEOF(*p), ""~*p); |
411 | 610 t.type = TOK.EOF; |
611 t.end = p; | |
612 tail = &t; | |
613 assert(t.start == t.end); | |
614 return; | |
615 } | |
616 | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
617 if (!isascii(c)) |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
618 { |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
619 c = decodeUTF8(); |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
620 if (isUniAlpha(c)) |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
621 goto Lidentifier; |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
622 } |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
623 |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
624 error(t.start, MID.IllegalCharacter, cast(dchar)c); |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
625 |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
626 ++p; |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
627 t.type = TOK.Illegal; |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
628 t.dchar_ = c; |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
629 t.end = p; |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
630 return; |
4 | 631 } |
3 | 632 } |
4 | 633 |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
634 template toUint(char[] T) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
635 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
636 static assert(0 < T.length && T.length <= 4); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
637 static if (T.length == 1) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
638 const uint toUint = T[0]; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
639 else |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
640 const uint toUint = (T[0] << ((T.length-1)*8)) | toUint!(T[1..$]); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
641 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
642 static assert(toUint!("\xAA\xBB\xCC\xDD") == 0xAABBCCDD); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
643 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
644 // Can't use this yet due to a bug in DMD (bug id=1534). |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
645 template case_(char[] str, TOK tok, char[] label) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
646 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
647 const char[] case_ = |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
648 `case `~toUint!(str).stringof~`: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
649 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
650 goto `~label~`;`; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
651 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
652 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
653 template case_L4(char[] str, TOK tok) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
654 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
655 const char[] case_L4 = case_!(str, tok, "Lcommon_4"); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
656 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
657 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
658 template case_L3(char[] str, TOK tok) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
659 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
660 const char[] case_L3 = case_!(str, tok, "Lcommon_3"); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
661 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
662 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
663 template case_L2(char[] str, TOK tok) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
664 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
665 const char[] case_L2 = case_!(str, tok, "Lcommon_2"); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
666 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
667 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
668 template case_L1(char[] str, TOK tok) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
669 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
670 const char[] case_L3 = case_!(str, tok, "Lcommon"); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
671 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
672 |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
673 public void scan_(ref Token t) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
674 in |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
675 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
676 assert(text.ptr <= p && p < end); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
677 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
678 out |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
679 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
680 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type)); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
681 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type)); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
682 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
683 body |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
684 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
685 // Scan whitespace. |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
686 if (isspace(*p)) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
687 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
688 t.ws = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
689 while (isspace(*++p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
690 {} |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
691 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
692 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
693 // Scan a token. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
694 t.start = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
695 // Newline. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
696 switch (*p) |
418
7354f15cd5e9
Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
416
diff
changeset
|
697 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
698 case '\r': |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
699 if (p[1] == '\n') |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
700 ++p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
701 case '\n': |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
702 assert(isNewlineEnd(p)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
703 ++p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
704 ++lineNum; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
705 setLineBegin(p); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
706 // this.newline = &t; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
707 t.type = TOK.Newline; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
708 t.filePath = this.errorPath; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
709 t.lineNum = lineNum; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
710 t.lineNum_hline = lineNum_hline; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
711 t.end = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
712 return; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
713 default: |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
714 if (isUnicodeNewline(p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
715 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
716 ++p; ++p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
717 goto case '\n'; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
718 } |
418
7354f15cd5e9
Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
416
diff
changeset
|
719 } |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
720 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
721 uint c = *p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
722 assert(end - p != 0); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
723 switch (end - p) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
724 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
725 case 1: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
726 goto L1character; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
727 case 2: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
728 c <<= 8; c |= p[1]; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
729 goto L2characters; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
730 case 3: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
731 c <<= 8; c |= p[1]; c <<= 8; c |= p[2]; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
732 goto L3characters; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
733 default: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
734 version(BigEndian) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
735 c = *cast(uint*)p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
736 else |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
737 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
738 c <<= 8; c |= p[1]; c <<= 8; c |= p[2]; c <<= 8; c |= p[3]; |
414
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
739 /+ |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
740 c = *cast(uint*)p; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
741 asm |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
742 { |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
743 mov EDX, c; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
744 bswap EDX; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
745 mov c, EDX; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
746 } |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
747 +/ |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
748 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
749 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
750 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
751 // 4 character tokens. |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
752 switch (c) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
753 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
754 case toUint!(">>>="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
755 t.type = TOK.RShiftAssign; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
756 goto Lcommon_4; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
757 case toUint!("!<>="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
758 t.type = TOK.Unordered; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
759 Lcommon_4: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
760 p += 4; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
761 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
762 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
763 default: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
764 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
765 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
766 c >>>= 8; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
767 L3characters: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
768 assert(p == t.start); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
769 // 3 character tokens. |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
770 switch (c) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
771 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
772 case toUint!(">>="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
773 t.type = TOK.RShiftAssign; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
774 goto Lcommon_3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
775 case toUint!(">>>"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
776 t.type = TOK.URShift; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
777 goto Lcommon_3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
778 case toUint!("<>="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
779 t.type = TOK.LorEorG; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
780 goto Lcommon_3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
781 case toUint!("<<="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
782 t.type = TOK.LShiftAssign; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
783 goto Lcommon_3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
784 case toUint!("!<="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
785 t.type = TOK.UorG; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
786 goto Lcommon_3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
787 case toUint!("!>="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
788 t.type = TOK.UorL; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
789 goto Lcommon_3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
790 case toUint!("!<>"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
791 t.type = TOK.UorE; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
792 goto Lcommon_3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
793 case toUint!("..."): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
794 t.type = TOK.Ellipses; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
795 Lcommon_3: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
796 p += 3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
797 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
798 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
799 default: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
800 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
801 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
802 c >>>= 8; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
803 L2characters: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
804 assert(p == t.start); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
805 // 2 character tokens. |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
806 switch (c) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
807 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
808 case toUint!("/+"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
809 ++p; // Skip / |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
810 return scanNestedComment(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
811 case toUint!("/*"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
812 ++p; // Skip / |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
813 return scanBlockComment(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
814 case toUint!("//"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
815 ++p; // Skip / |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
816 assert(*p == '/'); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
817 while (!isEndOfLine(++p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
818 isascii(*p) || decodeUTF8(); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
819 t.type = TOK.Comment; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
820 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
821 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
822 case toUint!(">="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
823 t.type = TOK.GreaterEqual; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
824 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
825 case toUint!(">>"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
826 t.type = TOK.RShift; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
827 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
828 case toUint!("<<"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
829 t.type = TOK.LShift; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
830 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
831 case toUint!("<="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
832 t.type = TOK.LessEqual; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
833 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
834 case toUint!("<>"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
835 t.type = TOK.LorG; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
836 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
837 case toUint!("!<"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
838 t.type = TOK.UorGorE; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
839 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
840 case toUint!("!>"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
841 t.type = TOK.UorLorE; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
842 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
843 case toUint!("!="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
844 t.type = TOK.NotEqual; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
845 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
846 case toUint!(".."): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
847 t.type = TOK.Slice; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
848 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
849 case toUint!("&&"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
850 t.type = TOK.AndLogical; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
851 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
852 case toUint!("&="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
853 t.type = TOK.AndAssign; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
854 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
855 case toUint!("||"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
856 t.type = TOK.OrLogical; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
857 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
858 case toUint!("|="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
859 t.type = TOK.OrAssign; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
860 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
861 case toUint!("++"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
862 t.type = TOK.PlusPlus; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
863 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
864 case toUint!("+="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
865 t.type = TOK.PlusAssign; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
866 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
867 case toUint!("--"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
868 t.type = TOK.MinusMinus; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
869 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
870 case toUint!("-="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
871 t.type = TOK.MinusAssign; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
872 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
873 case toUint!("=="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
874 t.type = TOK.Equal; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
875 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
876 case toUint!("~="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
877 t.type = TOK.CatAssign; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
878 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
879 case toUint!("*="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
880 t.type = TOK.MulAssign; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
881 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
882 case toUint!("/="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
883 t.type = TOK.DivAssign; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
884 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
885 case toUint!("^="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
886 t.type = TOK.XorAssign; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
887 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
888 case toUint!("%="): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
889 t.type = TOK.ModAssign; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
890 Lcommon_2: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
891 p += 2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
892 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
893 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
894 default: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
895 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
896 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
897 c >>>= 8; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
898 L1character: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
899 assert(p == t.start); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
900 assert(*p == c, Format("p={0},c={1}", *p, cast(dchar)c)); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
901 // 1 character tokens. |
414
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
902 // TODO: consider storing the token type in ptable. |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
903 switch (c) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
904 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
905 case '\'': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
906 return scanCharacterLiteral(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
907 case '`': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
908 return scanRawStringLiteral(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
909 case '"': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
910 return scanNormalStringLiteral(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
911 case '\\': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
912 char[] buffer; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
913 do |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
914 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
915 c = scanEscapeSequence(); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
916 if (isascii(c)) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
917 buffer ~= c; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
918 else |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
919 encodeUTF8(buffer, c); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
920 } while (*p == '\\') |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
921 buffer ~= 0; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
922 t.type = TOK.String; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
923 t.str = buffer; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
924 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
925 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
926 case '<': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
927 t.type = TOK.Greater; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
928 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
929 case '>': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
930 t.type = TOK.Less; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
931 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
932 case '^': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
933 t.type = TOK.Xor; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
934 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
935 case '!': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
936 t.type = TOK.Not; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
937 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
938 case '.': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
939 if (isdigit(p[1])) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
940 return scanReal(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
941 t.type = TOK.Dot; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
942 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
943 case '&': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
944 t.type = TOK.AndBinary; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
945 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
946 case '|': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
947 t.type = TOK.OrBinary; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
948 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
949 case '+': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
950 t.type = TOK.Plus; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
951 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
952 case '-': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
953 t.type = TOK.Minus; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
954 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
955 case '=': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
956 t.type = TOK.Assign; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
957 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
958 case '~': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
959 t.type = TOK.Tilde; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
960 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
961 case '*': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
962 t.type = TOK.Mul; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
963 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
964 case '/': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
965 t.type = TOK.Div; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
966 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
967 case '%': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
968 t.type = TOK.Mod; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
969 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
970 case '(': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
971 t.type = TOK.LParen; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
972 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
973 case ')': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
974 t.type = TOK.RParen; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
975 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
976 case '[': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
977 t.type = TOK.LBracket; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
978 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
979 case ']': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
980 t.type = TOK.RBracket; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
981 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
982 case '{': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
983 t.type = TOK.LBrace; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
984 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
985 case '}': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
986 t.type = TOK.RBrace; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
987 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
988 case ':': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
989 t.type = TOK.Colon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
990 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
991 case ';': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
992 t.type = TOK.Semicolon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
993 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
994 case '?': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
995 t.type = TOK.Question; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
996 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
997 case ',': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
998 t.type = TOK.Comma; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
999 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1000 case '$': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1001 t.type = TOK.Dollar; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1002 Lcommon: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1003 ++p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1004 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1005 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1006 case '#': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1007 return scanSpecialTokenSequence(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1008 default: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1009 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1010 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1011 assert(p == t.start); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1012 assert(*p == c); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1013 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1014 // TODO: consider moving isidbeg() and isdigit() up. |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1015 if (isidbeg(c)) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1016 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1017 if (c == 'r' && p[1] == '"' && ++p) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1018 return scanRawStringLiteral(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1019 if (c == 'x' && p[1] == '"') |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1020 return scanHexStringLiteral(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1021 version(D2) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1022 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1023 if (c == 'q' && p[1] == '"') |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1024 return scanDelimitedStringLiteral(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1025 if (c == 'q' && p[1] == '{') |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1026 return scanTokenStringLiteral(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1027 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1028 // Scan identifier. |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1029 Lidentifier: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1030 do |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1031 { c = *++p; } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1032 while (isident(c) || !isascii(c) && isUniAlpha(decodeUTF8())) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1033 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1034 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1035 |
502
4e14cd1b24da
Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
499
diff
changeset
|
1036 auto id = IdTable.lookup(t.srcText); |
499
52447db67938
Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
498
diff
changeset
|
1037 t.type = id.type; |
502
4e14cd1b24da
Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
499
diff
changeset
|
1038 t.ident = id; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1039 |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
1040 if (t.type == TOK.Identifier || t.isKeyword) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1041 return; |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
1042 else if (t.isSpecialToken) |
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
1043 finalizeSpecialToken(t); |
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
1044 else if (t.type == TOK.EOF) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1045 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1046 tail = &t; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1047 assert(t.srcText == "__EOF__"); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1048 } |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
1049 else |
502
4e14cd1b24da
Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
499
diff
changeset
|
1050 assert(0, "unexpected token type: " ~ Token.toString(t.type)); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1051 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1052 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1053 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1054 if (isdigit(c)) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1055 return scanNumber(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1056 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1057 // Check for EOF |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
1058 if (isEOF(c)) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1059 { |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
1060 assert(isEOF(*p), *p~""); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1061 t.type = TOK.EOF; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1062 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1063 tail = &t; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1064 assert(t.start == t.end); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1065 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1066 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1067 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1068 if (!isascii(c)) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1069 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1070 c = decodeUTF8(); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1071 if (isUniAlpha(c)) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1072 goto Lidentifier; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1073 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1074 |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1075 error(t.start, MID.IllegalCharacter, cast(dchar)c); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1076 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1077 ++p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1078 t.type = TOK.Illegal; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1079 t.dchar_ = c; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1080 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1081 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1082 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1083 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1084 void scanBlockComment(ref Token t) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1085 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1086 assert(p[-1] == '/' && *p == '*'); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1087 auto tokenLineNum = lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1088 auto tokenLineBegin = lineBegin; |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1089 Loop: |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1090 while (1) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1091 { |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1092 switch (*++p) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1093 { |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1094 case '*': |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1095 if (p[1] != '/') |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1096 continue; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1097 p += 2; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1098 break Loop; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1099 case '\r': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1100 if (p[1] == '\n') |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1101 ++p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1102 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1103 assert(isNewlineEnd(p)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1104 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1105 setLineBegin(p+1); |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1106 break; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1107 default: |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1108 if (!isascii(*p)) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1109 { |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1110 if (isUnicodeNewlineChar(decodeUTF8())) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1111 goto case '\n'; |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1112 } |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1113 else if (isEOF(*p)) |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1114 { |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1115 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedBlockComment); |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1116 break Loop; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1117 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1118 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1119 } |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1120 t.type = TOK.Comment; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1121 t.end = p; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1122 return; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1123 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1124 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1125 void scanNestedComment(ref Token t) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1126 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1127 assert(p[-1] == '/' && *p == '+'); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1128 auto tokenLineNum = lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1129 auto tokenLineBegin = lineBegin; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1130 uint level = 1; |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1131 Loop: |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1132 while (1) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1133 { |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1134 switch (*++p) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1135 { |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1136 case '/': |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1137 if (p[1] == '+') |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1138 ++p, ++level; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1139 continue; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1140 case '+': |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1141 if (p[1] != '/') |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1142 continue; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1143 ++p; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1144 if (--level != 0) |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1145 continue; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1146 ++p; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1147 break Loop; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1148 case '\r': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1149 if (p[1] == '\n') |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1150 ++p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1151 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1152 assert(isNewlineEnd(p)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1153 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1154 setLineBegin(p+1); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1155 continue; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1156 default: |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1157 if (!isascii(*p)) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1158 { |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1159 if (isUnicodeNewlineChar(decodeUTF8())) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1160 goto case '\n'; |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1161 } |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1162 else if (isEOF(*p)) |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1163 { |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1164 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedNestedComment); |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1165 break Loop; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1166 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1167 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1168 } |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1169 t.type = TOK.Comment; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1170 t.end = p; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1171 return; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1172 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1173 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1174 char scanPostfix() |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1175 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1176 assert(p[-1] == '"' || p[-1] == '`' || |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1177 { version(D2) return p[-1] == '}'; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1178 else return 0; }() |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1179 ); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1180 switch (*p) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1181 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1182 case 'c': |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1183 case 'w': |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1184 case 'd': |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1185 return *p++; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1186 default: |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1187 return 0; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1188 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1189 assert(0); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1190 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1191 |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1192 void scanNormalStringLiteral(ref Token t) |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1193 { |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1194 assert(*p == '"'); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1195 auto tokenLineNum = lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1196 auto tokenLineBegin = lineBegin; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1197 t.type = TOK.String; |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1198 char[] buffer; |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1199 uint c; |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1200 while (1) |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1201 { |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1202 c = *++p; |
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1203 switch (c) |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1204 { |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1205 case '"': |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1206 ++p; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1207 t.pf = scanPostfix(); |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1208 Lreturn: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1209 t.str = buffer ~ '\0'; |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1210 t.end = p; |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1211 return; |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1212 case '\\': |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1213 c = scanEscapeSequence(); |
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1214 --p; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1215 if (isascii(c)) |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1216 break; |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
1217 encodeUTF8(buffer, c); |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1218 continue; |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1219 case '\r': |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1220 if (p[1] == '\n') |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1221 ++p; |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1222 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1223 assert(isNewlineEnd(p)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1224 c = '\n'; // Convert Newline to \n. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1225 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1226 setLineBegin(p+1); |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1227 break; |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1228 case 0, _Z_: |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1229 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedString); |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1230 goto Lreturn; |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1231 default: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1232 if (!isascii(c)) |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1233 { |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1234 c = decodeUTF8(); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1235 if (isUnicodeNewlineChar(c)) |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1236 goto case '\n'; |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1237 encodeUTF8(buffer, c); |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1238 continue; |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1239 } |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1240 } |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
1241 assert(isascii(c)); |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1242 buffer ~= c; |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1243 } |
67
996065105910
- Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents:
66
diff
changeset
|
1244 assert(0); |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1245 } |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1246 |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1247 void scanCharacterLiteral(ref Token t) |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1248 { |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1249 assert(*p == '\''); |
46 | 1250 ++p; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1251 t.type = TOK.CharLiteral; |
46 | 1252 switch (*p) |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1253 { |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1254 case '\\': |
46 | 1255 t.dchar_ = scanEscapeSequence(); |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1256 break; |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1257 case '\'': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1258 error(t.start, MID.EmptyCharacterLiteral); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1259 break; |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1260 default: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1261 if (isEndOfLine(p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1262 break; |
46 | 1263 uint c = *p; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1264 if (!isascii(c)) |
49 | 1265 c = decodeUTF8(); |
46 | 1266 t.dchar_ = c; |
1267 ++p; | |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1268 } |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1269 |
46 | 1270 if (*p == '\'') |
1271 ++p; | |
1272 else | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1273 error(t.start, MID.UnterminatedCharacterLiteral); |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1274 t.end = p; |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1275 } |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1276 |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1277 void scanRawStringLiteral(ref Token t) |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1278 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1279 assert(*p == '`' || *p == '"' && p[-1] == 'r'); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1280 auto tokenLineNum = lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1281 auto tokenLineBegin = lineBegin; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1282 t.type = TOK.String; |
33 | 1283 uint delim = *p; |
1284 char[] buffer; | |
1285 uint c; | |
1286 while (1) | |
1287 { | |
1288 c = *++p; | |
1289 switch (c) | |
1290 { | |
1291 case '\r': | |
1292 if (p[1] == '\n') | |
1293 ++p; | |
1294 case '\n': | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1295 assert(isNewlineEnd(p)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1296 c = '\n'; // Convert Newline to '\n'. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1297 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1298 setLineBegin(p+1); |
52
f65a83c27638
- Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well.
aziz
parents:
51
diff
changeset
|
1299 break; |
33 | 1300 case '`': |
1301 case '"': | |
1302 if (c == delim) | |
1303 { | |
1304 ++p; | |
1305 t.pf = scanPostfix(); | |
1306 Lreturn: | |
1307 t.str = buffer ~ '\0'; | |
1308 t.end = p; | |
1309 return; | |
1310 } | |
1311 break; | |
1312 case 0, _Z_: | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1313 error(tokenLineNum, tokenLineBegin, t.start, |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1314 delim == 'r' ? MID.UnterminatedRawString : MID.UnterminatedBackQuoteString); |
33 | 1315 goto Lreturn; |
1316 default: | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1317 if (!isascii(c)) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1318 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1319 c = decodeUTF8(); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1320 if (isUnicodeNewlineChar(c)) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1321 goto case '\n'; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1322 encodeUTF8(buffer, c); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1323 continue; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1324 } |
33 | 1325 } |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
1326 assert(isascii(c)); |
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
1327 buffer ~= c; |
33 | 1328 } |
1329 assert(0); | |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1330 } |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1331 |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1332 void scanHexStringLiteral(ref Token t) |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1333 { |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1334 assert(p[0] == 'x' && p[1] == '"'); |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1335 t.type = TOK.String; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1336 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1337 auto tokenLineNum = lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1338 auto tokenLineBegin = lineBegin; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1339 |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1340 uint c; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1341 ubyte[] buffer; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1342 ubyte h; // hex number |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1343 uint n; // number of hex digits |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1344 |
53
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1345 ++p; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1346 assert(*p == '"'); |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1347 while (1) |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1348 { |
53
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1349 c = *++p; |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1350 switch (c) |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1351 { |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1352 case '"': |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1353 if (n & 1) |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1354 error(tokenLineNum, tokenLineBegin, t.start, MID.OddNumberOfDigitsInHexString); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1355 ++p; |
53
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1356 t.pf = scanPostfix(); |
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1357 Lreturn: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1358 t.str = cast(string) (buffer ~= 0); |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1359 t.end = p; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1360 return; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1361 case '\r': |
53
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1362 if (p[1] == '\n') |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1363 ++p; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1364 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1365 assert(isNewlineEnd(p)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1366 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1367 setLineBegin(p+1); |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1368 continue; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1369 default: |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1370 if (ishexad(c)) |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1371 { |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1372 if (c <= '9') |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1373 c -= '0'; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1374 else if (c <= 'F') |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1375 c -= 'A' - 10; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1376 else |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1377 c -= 'a' - 10; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1378 |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1379 if (n & 1) |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1380 { |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1381 h <<= 4; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1382 h |= c; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1383 buffer ~= h; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1384 } |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1385 else |
112 | 1386 h = cast(ubyte)c; |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1387 ++n; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1388 continue; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1389 } |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1390 else if (isspace(c)) |
419
89e40d43065d
Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
418
diff
changeset
|
1391 continue; // Skip spaces. |
496
5a607597dc22
Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
494
diff
changeset
|
1392 else if (isEOF(c)) |
53
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1393 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1394 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedHexString); |
53
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1395 t.pf = 0; |
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1396 goto Lreturn; |
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1397 } |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1398 else |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1399 { |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1400 auto errorAt = p; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1401 if (!isascii(c)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1402 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1403 c = decodeUTF8(); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1404 if (isUnicodeNewlineChar(c)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1405 goto case '\n'; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1406 } |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1407 error(errorAt, MID.NonHexCharInHexString, cast(dchar)c); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1408 } |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1409 } |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1410 } |
53
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1411 assert(0); |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1412 } |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1413 |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1414 version(D2) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1415 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1416 void scanDelimitedStringLiteral(ref Token t) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1417 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1418 assert(p[0] == 'q' && p[1] == '"'); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1419 t.type = TOK.String; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1420 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1421 auto tokenLineNum = lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1422 auto tokenLineBegin = lineBegin; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1423 |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1424 char[] buffer; |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1425 dchar opening_delim = 0, // 0 if no nested delimiter or '[', '(', '<', '{' |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1426 closing_delim; // Will be ']', ')', '>', '}, |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1427 // the first character of an identifier or |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1428 // any other Unicode/ASCII character. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1429 char[] str_delim; // Identifier delimiter. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1430 uint level = 1; // Counter for nestable delimiters. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1431 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1432 ++p; ++p; // Skip q" |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1433 uint c = *p; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1434 switch (c) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1435 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1436 case '(': |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1437 opening_delim = c; |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1438 closing_delim = ')'; // c + 1 |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1439 break; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1440 case '[', '<', '{': |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1441 opening_delim = c; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1442 closing_delim = c + 2; // Get to closing counterpart. Feature of ASCII table. |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1443 break; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1444 default: |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1445 dchar scanNewline() |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1446 { |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1447 switch (*p) |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1448 { |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1449 case '\r': |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1450 if (p[1] == '\n') |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1451 ++p; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1452 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1453 assert(isNewlineEnd(p)); |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1454 ++p; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1455 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1456 setLineBegin(p); |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1457 return '\n'; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1458 default: |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1459 if (isUnicodeNewline(p)) |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1460 { |
419
89e40d43065d
Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
418
diff
changeset
|
1461 ++p; ++p; |
89e40d43065d
Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
418
diff
changeset
|
1462 goto case '\n'; |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1463 } |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1464 } |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1465 return 0; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1466 } |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1467 // Skip leading newlines: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1468 while (scanNewline() != 0) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1469 {} |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1470 assert(!isNewline(p)); |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1471 |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1472 char* begin = p; |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1473 c = *p; |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1474 closing_delim = c; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1475 // TODO: Check for non-printable characters? |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1476 if (!isascii(c)) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1477 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1478 closing_delim = decodeUTF8(); |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1479 if (!isUniAlpha(closing_delim)) |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1480 break; // Not an identifier. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1481 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1482 else if (!isidbeg(c)) |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1483 break; // Not an identifier. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1484 |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1485 // Parse Identifier + EndOfLine |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1486 do |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1487 { c = *++p; } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1488 while (isident(c) || !isascii(c) && isUniAlpha(decodeUTF8())) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1489 // Store identifier |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1490 str_delim = begin[0..p-begin]; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1491 // Scan newline |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1492 if (scanNewline() == '\n') |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1493 --p; // Go back one because of "c = *++p;" in main loop. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1494 else |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1495 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1496 // TODO: error(p, MID.ExpectedNewlineAfterIdentDelim); |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1497 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1498 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1499 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1500 bool checkStringDelim(char* p) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1501 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1502 assert(str_delim.length != 0); |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1503 if (buffer[$-1] == '\n' && // Last character copied to buffer must be '\n'. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1504 end-p >= str_delim.length && // Check remaining length. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1505 p[0..str_delim.length] == str_delim) // Compare. |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1506 return true; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1507 return false; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1508 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1509 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1510 while (1) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1511 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1512 c = *++p; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1513 switch (c) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1514 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1515 case '\r': |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1516 if (p[1] == '\n') |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1517 ++p; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1518 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1519 assert(isNewlineEnd(p)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1520 c = '\n'; // Convert Newline to '\n'. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1521 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1522 setLineBegin(p+1); |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1523 break; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1524 case 0, _Z_: |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1525 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedDelimitedString); |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1526 goto Lreturn3; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1527 default: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1528 if (!isascii(c)) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1529 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1530 auto begin = p; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1531 c = decodeUTF8(); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1532 if (isUnicodeNewlineChar(c)) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1533 goto case '\n'; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1534 if (c == closing_delim) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1535 { |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1536 if (str_delim.length) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1537 { |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1538 if (checkStringDelim(begin)) |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1539 { |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1540 p = begin + str_delim.length; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1541 goto Lreturn2; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1542 } |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1543 } |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1544 else |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1545 { |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1546 assert(level == 1); |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1547 --level; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1548 goto Lreturn; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1549 } |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1550 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1551 encodeUTF8(buffer, c); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1552 continue; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1553 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1554 else |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1555 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1556 if (c == opening_delim) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1557 ++level; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1558 else if (c == closing_delim) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1559 { |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1560 if (str_delim.length) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1561 { |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1562 if (checkStringDelim(p)) |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1563 { |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1564 p += str_delim.length; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1565 goto Lreturn2; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1566 } |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1567 } |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1568 else if (--level == 0) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1569 goto Lreturn; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1570 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1571 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1572 } |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
1573 assert(isascii(c)); |
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
1574 buffer ~= c; |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1575 } |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1576 Lreturn: // Character delimiter. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1577 assert(c == closing_delim); |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1578 assert(level == 0); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1579 ++p; // Skip closing delimiter. |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1580 Lreturn2: // String delimiter. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1581 if (*p == '"') |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1582 ++p; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1583 else |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1584 { |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1585 // TODO: error(p, MID.ExpectedDblQuoteAfterDelim, str_delim.length ? str_delim : closing_delim~""); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1586 } |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1587 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1588 t.pf = scanPostfix(); |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1589 Lreturn3: // Error. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1590 t.str = buffer ~ '\0'; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1591 t.end = p; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1592 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1593 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1594 void scanTokenStringLiteral(ref Token t) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1595 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1596 assert(p[0] == 'q' && p[1] == '{'); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1597 t.type = TOK.String; |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1598 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1599 auto tokenLineNum = lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1600 auto tokenLineBegin = lineBegin; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1601 |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1602 // A guard against changes to particular members: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1603 // this.lineNum_hline and this.errorPath |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1604 ++inTokenString; |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1605 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1606 uint lineNum = this.lineNum; |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1607 uint level = 1; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1608 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1609 ++p; ++p; // Skip q{ |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1610 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1611 auto prev_t = &t; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1612 Token* token; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1613 while (1) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1614 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1615 token = new Token; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1616 scan(*token); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1617 // Save the tokens in a doubly linked list. |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1618 // Could be useful for various tools. |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1619 token.prev = prev_t; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1620 prev_t.next = token; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1621 prev_t = token; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1622 switch (token.type) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1623 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1624 case TOK.LBrace: |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1625 ++level; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1626 continue; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1627 case TOK.RBrace: |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1628 if (--level == 0) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1629 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1630 t.tok_str = t.next; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1631 t.next = null; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1632 break; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1633 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1634 continue; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1635 case TOK.EOF: |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1636 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedTokenString); |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1637 t.tok_str = t.next; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1638 t.next = token; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1639 break; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1640 default: |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1641 continue; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1642 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1643 break; // Exit loop. |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1644 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1645 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1646 assert(token.type == TOK.RBrace || token.type == TOK.EOF); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1647 assert(token.type == TOK.RBrace && t.next is null || |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1648 token.type == TOK.EOF && t.next !is null); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1649 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1650 char[] buffer; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1651 // token points to } or EOF |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1652 if (token.type == TOK.EOF) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1653 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1654 t.end = token.start; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1655 buffer = t.srcText[2..$].dup ~ '\0'; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1656 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1657 else |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1658 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1659 // Assign to buffer before scanPostfix(). |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1660 t.end = p; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1661 buffer = t.srcText[2..$-1].dup ~ '\0'; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1662 t.pf = scanPostfix(); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1663 t.end = p; // Assign again because of postfix. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1664 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1665 // Convert newlines to '\n'. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1666 if (lineNum != this.lineNum) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1667 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1668 assert(buffer[$-1] == '\0'); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1669 uint i, j; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1670 for (; i < buffer.length; ++i) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1671 switch (buffer[i]) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1672 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1673 case '\r': |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1674 if (buffer[i+1] == '\n') |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1675 ++i; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1676 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1677 assert(isNewlineEnd(buffer.ptr + i)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1678 buffer[j++] = '\n'; // Convert Newline to '\n'. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1679 break; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1680 default: |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1681 if (isUnicodeNewline(buffer.ptr + i)) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1682 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1683 ++i; ++i; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1684 goto case '\n'; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1685 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1686 buffer[j++] = buffer[i]; // Copy. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1687 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1688 buffer.length = j; // Adjust length. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1689 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1690 assert(buffer[$-1] == '\0'); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1691 t.str = buffer; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1692 |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1693 --inTokenString; |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1694 } |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1695 } // version(D2) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1696 |
45 | 1697 dchar scanEscapeSequence() |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1698 out(result) |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
1699 { assert(isValidChar(result)); } |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1700 body |
45 | 1701 { |
386
392a0068fc61
Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
384
diff
changeset
|
1702 assert(*p == '\\'); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1703 |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1704 auto sequenceStart = p; // Used for error reporting. |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1705 |
386
392a0068fc61
Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
384
diff
changeset
|
1706 ++p; |
45 | 1707 uint c = char2ev(*p); |
82
fc645fb2fe72
- scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents:
71
diff
changeset
|
1708 if (c) |
fc645fb2fe72
- scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents:
71
diff
changeset
|
1709 { |
46 | 1710 ++p; |
45 | 1711 return c; |
46 | 1712 } |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1713 |
45 | 1714 uint digits = 2; |
1715 | |
1716 switch (*p) | |
1717 { | |
1718 case 'x': | |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1719 assert(c == 0); |
45 | 1720 while (1) |
1721 { | |
1722 ++p; | |
1723 if (ishexad(*p)) | |
1724 { | |
1725 c *= 16; | |
1726 if (*p <= '9') | |
48 | 1727 c += *p - '0'; |
45 | 1728 else if (*p <= 'F') |
48 | 1729 c += *p - 'A' + 10; |
45 | 1730 else |
48 | 1731 c += *p - 'a' + 10; |
82
fc645fb2fe72
- scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents:
71
diff
changeset
|
1732 |
fc645fb2fe72
- scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents:
71
diff
changeset
|
1733 if (!--digits) |
fc645fb2fe72
- scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents:
71
diff
changeset
|
1734 { |
48 | 1735 ++p; |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
1736 if (isValidChar(c)) |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1737 return c; // Return valid escape value. |
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1738 |
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1739 error(sequenceStart, MID.InvalidUnicodeEscapeSequence, sequenceStart[0..p-sequenceStart]); |
45 | 1740 break; |
48 | 1741 } |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1742 continue; |
45 | 1743 } |
432
3ead178e0662
Added message MID.InvalidUnicodeEscapeSequence.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
430
diff
changeset
|
1744 |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1745 error(sequenceStart, MID.InsufficientHexDigits); |
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1746 break; |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
1747 } |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1748 break; |
45 | 1749 case 'u': |
1750 digits = 4; | |
1751 goto case 'x'; | |
1752 case 'U': | |
1753 digits = 8; | |
1754 goto case 'x'; | |
1755 default: | |
48 | 1756 if (isoctal(*p)) |
45 | 1757 { |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1758 assert(c == 0); |
48 | 1759 c += *p - '0'; |
1760 ++p; | |
1761 if (!isoctal(*p)) | |
1762 return c; | |
1763 c *= 8; | |
1764 c += *p - '0'; | |
1765 ++p; | |
1766 if (!isoctal(*p)) | |
1767 return c; | |
1768 c *= 8; | |
1769 c += *p - '0'; | |
1770 ++p; | |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1771 return c; // Return valid escape value. |
48 | 1772 } |
1773 else if(*p == '&') | |
1774 { | |
1775 if (isalpha(*++p)) | |
45 | 1776 { |
272
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1777 auto begin = p; |
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1778 while (isalnum(*++p)) |
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1779 {} |
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1780 |
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1781 if (*p == ';') |
48 | 1782 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1783 // Pass entity excluding '&' and ';'. |
272
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1784 c = entity2Unicode(begin[0..p - begin]); |
386
392a0068fc61
Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
384
diff
changeset
|
1785 ++p; // Skip ; |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1786 if (c != 0xFFFF) |
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1787 return c; // Return valid escape value. |
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1788 else |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1789 error(sequenceStart, MID.UndefinedHTMLEntity, sequenceStart[0 .. p - sequenceStart]); |
272
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1790 } |
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1791 else |
432
3ead178e0662
Added message MID.InvalidUnicodeEscapeSequence.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
430
diff
changeset
|
1792 error(sequenceStart, MID.UnterminatedHTMLEntity, sequenceStart[0 .. p - sequenceStart]); |
45 | 1793 } |
48 | 1794 else |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1795 error(sequenceStart, MID.InvalidBeginHTMLEntity); |
45 | 1796 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1797 else if (isEndOfLine(p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1798 error(sequenceStart, MID.UndefinedEscapeSequence, |
496
5a607597dc22
Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
494
diff
changeset
|
1799 isEOF(*p) ? `\EOF` : `\NewLine`); |
45 | 1800 else |
386
392a0068fc61
Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
384
diff
changeset
|
1801 { |
392a0068fc61
Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
384
diff
changeset
|
1802 char[] str = `\`; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1803 if (isascii(c)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1804 str ~= *p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1805 else |
387
ad0cbd1c8881
Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
386
diff
changeset
|
1806 encodeUTF8(str, decodeUTF8()); |
ad0cbd1c8881
Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
386
diff
changeset
|
1807 ++p; |
ad0cbd1c8881
Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
386
diff
changeset
|
1808 // TODO: check for unprintable character? |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1809 error(sequenceStart, MID.UndefinedEscapeSequence, str); |
386
392a0068fc61
Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
384
diff
changeset
|
1810 } |
45 | 1811 } |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1812 return REPLACEMENT_CHAR; // Error: return replacement character. |
45 | 1813 } |
1814 | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1815 /* |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1816 IntegerLiteral:= (Dec|Hex|Bin|Oct)Suffix? |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1817 Dec:= (0|[1-9][0-9_]*) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1818 Hex:= 0[xX] HexDigits |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1819 Bin:= 0[bB][01_]+ |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1820 Oct:= 0[0-7_]+ |
68
7eb83dd38901
- Simplified suffix rule and added a few more numbers to unittest.
aziz
parents:
67
diff
changeset
|
1821 Suffix:= (L[uU]?|[uU]L?) |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1822 HexDigits:= [0-9a-zA-Z_]+ |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1823 |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1824 Invalid: "0b_", "0x_", "._" |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1825 */ |
15 | 1826 void scanNumber(ref Token t) |
1827 { | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1828 ulong ulong_; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1829 bool overflow; |
58 | 1830 bool isDecimal; |
57
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1831 size_t digits; |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1832 |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1833 if (*p != '0') |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1834 goto LscanInteger; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1835 ++p; // skip zero |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1836 // check for xX bB ... |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1837 switch (*p) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1838 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1839 case 'x','X': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1840 goto LscanHex; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1841 case 'b','B': |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1842 goto LscanBinary; |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1843 case 'L': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1844 if (p[1] == 'i') |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1845 goto LscanReal; // 0Li |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1846 break; // 0L |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1847 case '.': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1848 if (p[1] == '.') |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1849 break; // 0.. |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1850 // 0. |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1851 case 'i','f','F', // Imaginary and float literal suffixes. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1852 'e', 'E': // Float exponent. |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1853 goto LscanReal; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1854 default: |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1855 if (*p == '_') |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1856 goto LscanOctal; // 0_ |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1857 else if (isdigit(*p)) |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1858 { |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1859 if (*p == '8' || *p == '9') |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1860 goto Loctal_hasDecimalDigits; // 08 or 09 |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1861 else |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1862 goto Loctal_enter_loop; // 0[0-7] |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1863 } |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1864 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1865 |
323 | 1866 // Number 0 |
1867 assert(p[-1] == '0'); | |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1868 assert(*p != '_' && !isdigit(*p)); |
323 | 1869 assert(ulong_ == 0); |
58 | 1870 isDecimal = true; |
1871 goto Lfinalize; | |
1872 | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1873 LscanInteger: |
323 | 1874 assert(*p != 0 && isdigit(*p)); |
58 | 1875 isDecimal = true; |
323 | 1876 goto Lenter_loop_int; |
63 | 1877 while (1) |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1878 { |
63 | 1879 if (*++p == '_') |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1880 continue; |
63 | 1881 if (!isdigit(*p)) |
1882 break; | |
323 | 1883 Lenter_loop_int: |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1884 if (ulong_ < ulong.max/10 || (ulong_ == ulong.max/10 && *p <= '5')) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1885 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1886 ulong_ *= 10; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1887 ulong_ += *p - '0'; |
58 | 1888 continue; |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1889 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1890 // Overflow: skip following digits. |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1891 overflow = true; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1892 while (isdigit(*++p)) {} |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1893 break; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1894 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1895 |
61 | 1896 // The number could be a float, so check overflow below. |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1897 switch (*p) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1898 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1899 case '.': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1900 if (p[1] != '.') |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1901 goto LscanReal; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1902 break; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1903 case 'L': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1904 if (p[1] != 'i') |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1905 break; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1906 case 'i', 'f', 'F', 'e', 'E': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1907 goto LscanReal; |
58 | 1908 default: |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1909 } |
58 | 1910 |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1911 if (overflow) |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1912 error(t.start, MID.OverflowDecimalNumber); |
58 | 1913 |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1914 assert((isdigit(p[-1]) || p[-1] == '_') && !isdigit(*p) && *p != '_'); |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1915 goto Lfinalize; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1916 |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1917 LscanHex: |
61 | 1918 assert(digits == 0); |
410
4d9ee8e60712
Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
403
diff
changeset
|
1919 assert(*p == 'x' || *p == 'X'); |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1920 while (1) |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1921 { |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1922 if (*++p == '_') |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1923 continue; |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1924 if (!ishexad(*p)) |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1925 break; |
61 | 1926 ++digits; |
57
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1927 ulong_ *= 16; |
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1928 if (*p <= '9') |
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1929 ulong_ += *p - '0'; |
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1930 else if (*p <= 'F') |
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1931 ulong_ += *p - 'A' + 10; |
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1932 else |
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1933 ulong_ += *p - 'a' + 10; |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1934 } |
57
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1935 |
410
4d9ee8e60712
Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
403
diff
changeset
|
1936 assert(ishexad(p[-1]) || p[-1] == '_' || p[-1] == 'x' || p[-1] == 'X'); |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
1937 assert(!ishexad(*p) && *p != '_'); |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
1938 |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1939 switch (*p) |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1940 { |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1941 case '.': |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
1942 if (p[1] == '.') |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1943 break; |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
1944 case 'p', 'P': |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
1945 return scanHexReal(t); |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1946 default: |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1947 } |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
1948 |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1949 if (digits == 0 || digits > 16) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1950 error(t.start, digits == 0 ? MID.NoDigitsInHexNumber : MID.OverflowHexNumber); |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
1951 |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1952 goto Lfinalize; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1953 |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1954 LscanBinary: |
57
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1955 assert(digits == 0); |
410
4d9ee8e60712
Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
403
diff
changeset
|
1956 assert(*p == 'b' || *p == 'B'); |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1957 while (1) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1958 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1959 if (*++p == '0') |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1960 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1961 ++digits; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1962 ulong_ *= 2; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1963 } |
367
dda55fae37de
- ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents:
365
diff
changeset
|
1964 else if (*p == '1') |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1965 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1966 ++digits; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1967 ulong_ *= 2; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1968 ulong_ += *p - '0'; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1969 } |
367
dda55fae37de
- ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents:
365
diff
changeset
|
1970 else if (*p == '_') |
dda55fae37de
- ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents:
365
diff
changeset
|
1971 continue; |
dda55fae37de
- ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents:
365
diff
changeset
|
1972 else |
dda55fae37de
- ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents:
365
diff
changeset
|
1973 break; |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1974 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1975 |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1976 if (digits == 0 || digits > 64) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1977 error(t.start, digits == 0 ? MID.NoDigitsInBinNumber : MID.OverflowBinaryNumber); |
59
3e594725899a
- Issuing error when no digits were found in hex and binary numbers.
aziz
parents:
58
diff
changeset
|
1978 |
410
4d9ee8e60712
Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
403
diff
changeset
|
1979 assert(p[-1] == '0' || p[-1] == '1' || p[-1] == '_' || p[-1] == 'b' || p[-1] == 'B', p[-1] ~ ""); |
367
dda55fae37de
- ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents:
365
diff
changeset
|
1980 assert( !(*p == '0' || *p == '1' || *p == '_') ); |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1981 goto Lfinalize; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1982 |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1983 LscanOctal: |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1984 assert(*p == '_'); |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1985 while (1) |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1986 { |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1987 if (*++p == '_') |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1988 continue; |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1989 if (!isoctal(*p)) |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1990 break; |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1991 Loctal_enter_loop: |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1992 if (ulong_ < ulong.max/2 || (ulong_ == ulong.max/2 && *p <= '1')) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1993 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1994 ulong_ *= 8; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1995 ulong_ += *p - '0'; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1996 continue; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1997 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1998 // Overflow: skip following digits. |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1999 overflow = true; |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
2000 while (isoctal(*++p)) {} |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2001 break; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2002 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2003 |
63 | 2004 bool hasDecimalDigits; |
2005 if (isdigit(*p)) | |
2006 { | |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
2007 Loctal_hasDecimalDigits: |
63 | 2008 hasDecimalDigits = true; |
2009 while (isdigit(*++p)) {} | |
2010 } | |
2011 | |
2012 // The number could be a float, so check errors below. | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2013 switch (*p) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2014 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2015 case '.': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2016 if (p[1] != '.') |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2017 goto LscanReal; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2018 break; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2019 case 'L': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2020 if (p[1] != 'i') |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2021 break; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2022 case 'i', 'f', 'F', 'e', 'E': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2023 goto LscanReal; |
63 | 2024 default: |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2025 } |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2026 |
63 | 2027 if (hasDecimalDigits) |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2028 error(t.start, MID.OctalNumberHasDecimals); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2029 |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2030 if (overflow) |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2031 error(t.start, MID.OverflowOctalNumber); |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2032 // goto Lfinalize; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2033 |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2034 Lfinalize: |
58 | 2035 enum Suffix |
2036 { | |
2037 None = 0, | |
2038 Unsigned = 1, | |
60 | 2039 Long = 2 |
58 | 2040 } |
60 | 2041 |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2042 // Scan optional suffix: L, Lu, LU, u, uL, U or UL. |
58 | 2043 Suffix suffix; |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2044 while (1) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2045 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2046 switch (*p) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2047 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2048 case 'L': |
58 | 2049 if (suffix & Suffix.Long) |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2050 break; |
60 | 2051 suffix |= Suffix.Long; |
2052 ++p; | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2053 continue; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2054 case 'u', 'U': |
58 | 2055 if (suffix & Suffix.Unsigned) |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2056 break; |
60 | 2057 suffix |= Suffix.Unsigned; |
2058 ++p; | |
2059 continue; | |
58 | 2060 default: |
2061 break; | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2062 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2063 break; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2064 } |
58 | 2065 |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2066 // Determine type of Integer. |
58 | 2067 switch (suffix) |
2068 { | |
2069 case Suffix.None: | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2070 if (ulong_ & 0x8000_0000_0000_0000) |
58 | 2071 { |
2072 if (isDecimal) | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2073 error(t.start, MID.OverflowDecimalSign); |
58 | 2074 t.type = TOK.Uint64; |
2075 } | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2076 else if (ulong_ & 0xFFFF_FFFF_0000_0000) |
58 | 2077 t.type = TOK.Int64; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2078 else if (ulong_ & 0x8000_0000) |
58 | 2079 t.type = isDecimal ? TOK.Int64 : TOK.Uint32; |
2080 else | |
2081 t.type = TOK.Int32; | |
2082 break; | |
2083 case Suffix.Unsigned: | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2084 if (ulong_ & 0xFFFF_FFFF_0000_0000) |
58 | 2085 t.type = TOK.Uint64; |
2086 else | |
2087 t.type = TOK.Uint32; | |
2088 break; | |
2089 case Suffix.Long: | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2090 if (ulong_ & 0x8000_0000_0000_0000) |
58 | 2091 { |
2092 if (isDecimal) | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2093 error(t.start, MID.OverflowDecimalSign); |
58 | 2094 t.type = TOK.Uint64; |
2095 } | |
2096 else | |
2097 t.type = TOK.Int64; | |
2098 break; | |
2099 case Suffix.Unsigned | Suffix.Long: | |
2100 t.type = TOK.Uint64; | |
2101 break; | |
2102 default: | |
2103 assert(0); | |
2104 } | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2105 t.ulong_ = ulong_; |
15 | 2106 t.end = p; |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2107 return; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2108 LscanReal: |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2109 scanReal(t); |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2110 return; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2111 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2112 |
63 | 2113 /* |
2114 FloatLiteral:= Float[fFL]?i? | |
2115 Float:= DecFloat | HexFloat | |
67
996065105910
- Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents:
66
diff
changeset
|
2116 DecFloat:= ([0-9][0-9_]*[.][0-9_]*DecExponent?) | [.][0-9][0-9_]*DecExponent? | [0-9][0-9_]*DecExponent |
63 | 2117 DecExponent:= [eE][+-]?[0-9][0-9_]* |
2118 HexFloat:= 0[xX](HexDigits[.]HexDigits | [.][0-9a-zA-Z]HexDigits? | HexDigits)HexExponent | |
2119 HexExponent:= [pP][+-]?[0-9][0-9_]* | |
2120 */ | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2121 void scanReal(ref Token t) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2122 { |
63 | 2123 if (*p == '.') |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2124 { |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2125 assert(p[1] != '.'); |
63 | 2126 // This function was called by scan() or scanNumber(). |
2127 while (isdigit(*++p) || *p == '_') {} | |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2128 } |
63 | 2129 else |
2130 // This function was called by scanNumber(). | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2131 assert(delegate () |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2132 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2133 switch (*p) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2134 { |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2135 case 'L': |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2136 if (p[1] != 'i') |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2137 return false; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2138 case 'i', 'f', 'F', 'e', 'E': |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2139 return true; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2140 default: |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2141 } |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2142 return false; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2143 }() |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2144 ); |
63 | 2145 |
2146 // Scan exponent. | |
2147 if (*p == 'e' || *p == 'E') | |
2148 { | |
2149 ++p; | |
2150 if (*p == '-' || *p == '+') | |
2151 ++p; | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2152 if (isdigit(*p)) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2153 while (isdigit(*++p) || *p == '_') {} |
63 | 2154 else |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2155 error(t.start, MID.FloatExpMustStartWithDigit); |
63 | 2156 } |
2157 | |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2158 // Copy whole number and remove underscores from buffer. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2159 char[] buffer = t.start[0..p-t.start].dup; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2160 uint j; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2161 foreach (c; buffer) |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2162 if (c != '_') |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2163 buffer[j++] = c; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2164 buffer.length = j; // Adjust length. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2165 buffer ~= 0; // Terminate for C functions. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2166 |
63 | 2167 finalizeFloat(t, buffer); |
30 | 2168 } |
2169 | |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2170 void scanHexReal(ref Token t) |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2171 { |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2172 assert(*p == '.' || *p == 'p' || *p == 'P'); |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2173 MID mid; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2174 if (*p == '.') |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2175 while (ishexad(*++p) || *p == '_') |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2176 {} |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2177 // Decimal exponent is required. |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2178 if (*p != 'p' && *p != 'P') |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2179 { |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2180 mid = MID.HexFloatExponentRequired; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2181 goto Lerr; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2182 } |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2183 // Scan exponent |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2184 assert(*p == 'p' || *p == 'P'); |
393
fce1e6133dac
Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
392
diff
changeset
|
2185 ++p; |
fce1e6133dac
Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
392
diff
changeset
|
2186 if (*p == '+' || *p == '-') |
fce1e6133dac
Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
392
diff
changeset
|
2187 ++p; |
fce1e6133dac
Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
392
diff
changeset
|
2188 if (!isdigit(*p)) |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2189 { |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2190 mid = MID.HexFloatExpMustStartWithDigit; |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2191 goto Lerr; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2192 } |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2193 while (isdigit(*++p) || *p == '_') |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2194 {} |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2195 // Copy whole number and remove underscores from buffer. |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2196 char[] buffer = t.start[0..p-t.start].dup; |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2197 uint j; |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2198 foreach (c; buffer) |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2199 if (c != '_') |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2200 buffer[j++] = c; |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2201 buffer.length = j; // Adjust length. |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2202 buffer ~= 0; // Terminate for C functions. |
63 | 2203 finalizeFloat(t, buffer); |
2204 return; | |
2205 Lerr: | |
2206 t.type = TOK.Float32; | |
2207 t.end = p; | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2208 error(t.start, mid); |
63 | 2209 } |
2210 | |
2211 void finalizeFloat(ref Token t, string buffer) | |
2212 { | |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2213 assert(buffer[$-1] == 0); |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2214 // Float number is well-formed. Check suffixes and do conversion. |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2215 switch (*p) |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2216 { |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2217 case 'f', 'F': |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2218 t.type = TOK.Float32; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2219 t.float_ = strtof(buffer.ptr, null); |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2220 ++p; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2221 break; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2222 case 'L': |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2223 t.type = TOK.Float80; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2224 t.real_ = strtold(buffer.ptr, null); |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2225 ++p; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2226 break; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2227 default: |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2228 t.type = TOK.Float64; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2229 t.double_ = strtod(buffer.ptr, null); |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2230 break; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2231 } |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2232 if (*p == 'i') |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2233 { |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2234 ++p; |
63 | 2235 t.type += 3; // Switch to imaginary counterpart. |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2236 } |
391
33b566df6af4
Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
390
diff
changeset
|
2237 if (errno() == ERANGE) |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2238 error(t.start, MID.OverflowFloatNumber); |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2239 t.end = p; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2240 } |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2241 |
30 | 2242 /// Scan special token: #line Integer [Filespec] EndOfLine |
360
b6a3755eba94
- Renamed scanSpecialToken() to scanSpecialTokenSequence().
aziz
parents:
350
diff
changeset
|
2243 void scanSpecialTokenSequence(ref Token t) |
30 | 2244 { |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
2245 assert(*p == '#'); |
323 | 2246 t.type = TOK.HashLine; |
2247 | |
30 | 2248 MID mid; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2249 auto errorAtColumn = p; |
30 | 2250 |
323 | 2251 ++p; |
2252 if (p[0] != 'l' || p[1] != 'i' || p[2] != 'n' || p[3] != 'e') | |
51 | 2253 { |
370 | 2254 mid = MID.ExpectedIdentifierSTLine; |
51 | 2255 goto Lerr; |
2256 } | |
323 | 2257 p += 3; |
2258 | |
388
ae154eceba65
Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
387
diff
changeset
|
2259 // TODO: #line58"path/file" is legal. Require spaces? |
ae154eceba65
Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
387
diff
changeset
|
2260 // State.Space could be used for that purpose. |
323 | 2261 enum State |
388
ae154eceba65
Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
387
diff
changeset
|
2262 { /+Space,+/ Integer, Filespec, End } |
51 | 2263 |
388
ae154eceba65
Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
387
diff
changeset
|
2264 State state = State.Integer; |
323 | 2265 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2266 while (!isEndOfLine(++p)) |
51 | 2267 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2268 if (isspace(*p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2269 continue; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2270 if (state == State.Integer) |
323 | 2271 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2272 if (!isdigit(*p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2273 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2274 errorAtColumn = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2275 mid = MID.ExpectedIntegerAfterSTLine; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2276 goto Lerr; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2277 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2278 t.tokLineNum = new Token; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2279 scan(*t.tokLineNum); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2280 if (t.tokLineNum.type != TOK.Int32 && t.tokLineNum.type != TOK.Uint32) |
323 | 2281 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2282 errorAtColumn = t.tokLineNum.start; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2283 mid = MID.ExpectedIntegerAfterSTLine; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2284 goto Lerr; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2285 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2286 --p; // Go one back because scan() advanced p past the integer. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2287 state = State.Filespec; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2288 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2289 else if (state == State.Filespec) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2290 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2291 if (*p != '"') |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2292 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2293 errorAtColumn = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2294 mid = MID.ExpectedFilespec; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2295 goto Lerr; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2296 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2297 t.tokLineFilespec = new Token; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2298 t.tokLineFilespec.start = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2299 t.tokLineFilespec.type = TOK.Filespec; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2300 while (*++p != '"') |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2301 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2302 if (isEndOfLine(p)) |
323 | 2303 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2304 errorAtColumn = t.tokLineFilespec.start; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2305 mid = MID.UnterminatedFilespec; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2306 t.tokLineFilespec.end = p; |
323 | 2307 goto Lerr; |
2308 } | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2309 isascii(*p) || decodeUTF8(); |
323 | 2310 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2311 auto start = t.tokLineFilespec.start +1; // +1 skips '"' |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2312 t.tokLineFilespec.str = start[0 .. p - start]; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2313 t.tokLineFilespec.end = p + 1; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2314 state = State.End; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2315 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2316 else/+ if (state == State.End)+/ |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2317 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2318 mid = MID.UnterminatedSpecialToken; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2319 goto Lerr; |
323 | 2320 } |
51 | 2321 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2322 assert(isEndOfLine(p)); |
323 | 2323 |
388
ae154eceba65
Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
387
diff
changeset
|
2324 if (state == State.Integer) |
51 | 2325 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2326 errorAtColumn = p; |
388
ae154eceba65
Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
387
diff
changeset
|
2327 mid = MID.ExpectedIntegerAfterSTLine; |
30 | 2328 goto Lerr; |
2329 } | |
2330 | |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2331 // Evaluate #line only when not in token string. |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2332 if (!inTokenString && t.tokLineNum) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2333 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2334 this.lineNum_hline = this.lineNum - t.tokLineNum.uint_ + 1; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2335 if (t.tokLineFilespec) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2336 this.errorPath = t.tokLineFilespec.str; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2337 } |
323 | 2338 t.end = p; |
30 | 2339 |
2340 return; | |
2341 Lerr: | |
323 | 2342 t.end = p; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2343 error(errorAtColumn, mid); |
371 | 2344 } |
2345 | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2346 /++ |
414
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2347 Insert an empty dummy token before t. |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2348 Useful in the parsing phase for representing a node in the AST |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2349 that doesn't consume an actual token from the source text. |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2350 +/ |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2351 Token* insertEmptyTokenBefore(Token* t) |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2352 { |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2353 assert(t !is null && t.prev !is null); |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2354 assert(text.ptr <= t.start && t.start < end, Token.toString(t.type)); |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2355 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.type)); |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2356 |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2357 auto prev_t = t.prev; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2358 auto new_t = new Token; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2359 new_t.type = TOK.Empty; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2360 new_t.start = new_t.end = prev_t.end; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2361 // Link in new token. |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2362 prev_t.next = new_t; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2363 new_t.prev = prev_t; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2364 new_t.next = t; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2365 t.prev = new_t; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2366 return new_t; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2367 } |
105 | 2368 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2369 uint errorLineNumber(uint lineNum) |
239
7911f6a92e6e
- Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents:
223
diff
changeset
|
2370 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2371 return lineNum - this.lineNum_hline; |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
2372 } |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
2373 |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2374 void error(char* columnPos, MID mid, ...) |
207 | 2375 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2376 error_(this.lineNum, this.lineBegin, columnPos, mid, _arguments, _argptr); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2377 } |
207 | 2378 |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2379 void error(uint lineNum, char* lineBegin, char* columnPos, MID mid, ...) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2380 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2381 error_(lineNum, lineBegin, columnPos, mid, _arguments, _argptr); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2382 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2383 |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2384 void error_(uint lineNum, char* lineBegin, char* columnPos, MID mid, |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2385 TypeInfo[] _arguments, void* _argptr) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2386 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2387 lineNum = this.errorLineNumber(lineNum); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2388 auto location = new Location(errorPath, lineNum, lineBegin, columnPos); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2389 auto msg = Format(_arguments, _argptr, GetMsg(mid)); |
520
f203c5248d0b
Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
519
diff
changeset
|
2390 auto error = new LexerError(location, msg); |
f203c5248d0b
Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
519
diff
changeset
|
2391 errors ~= error; |
f203c5248d0b
Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
519
diff
changeset
|
2392 if (infoMan !is null) |
f203c5248d0b
Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
519
diff
changeset
|
2393 infoMan ~= error; |
207 | 2394 } |
2395 | |
306 | 2396 Token* getTokens() |
4 | 2397 { |
306 | 2398 while (nextToken() != TOK.EOF) |
2399 {} | |
2400 return head; | |
3 | 2401 } |
48 | 2402 |
487
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2403 /// Scan the whole text until EOF is encountered. |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2404 void scanAll() |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2405 { |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2406 while (nextToken() != TOK.EOF) |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2407 {} |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2408 } |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2409 |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2410 /// HEAD -> Newline -> First Token |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2411 Token* firstToken() |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2412 { |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2413 return this.head.next.next; |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2414 } |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2415 |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2416 static void loadKeywords(ref Identifier[string] table) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2417 { |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2418 foreach(k; keywords) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2419 table[k.str] = k; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2420 } |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2421 |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2422 /// Returns true if str is a valid D identifier. |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2423 static bool isIdentifierString(char[] str) |
365 | 2424 { |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2425 if (str.length == 0 || isdigit(str[0])) |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2426 return false; |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2427 size_t idx; |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2428 do |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2429 { |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2430 auto c = dil.Unicode.decode(str, idx); |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2431 if (c == ERROR_CHAR || !(isident(c) || !isascii(c) && isUniAlpha(c))) |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2432 return false; |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2433 } while (idx < str.length) |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2434 return true; |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2435 } |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2436 |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2437 /// Returns true if str is a keyword or a special token (__FILE__, __LINE__ etc.) |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2438 static bool isReservedIdentifier(char[] str) |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2439 { |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2440 if (str.length == 0) |
382
66477017cb95
Applied fixes to isNonReservedIdentifier().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
371
diff
changeset
|
2441 return false; |
66477017cb95
Applied fixes to isNonReservedIdentifier().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
371
diff
changeset
|
2442 |
365 | 2443 static Identifier[string] reserved_ids_table; |
2444 if (reserved_ids_table is null) | |
499
52447db67938
Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
498
diff
changeset
|
2445 Lexer.loadKeywords(reserved_ids_table); |
365 | 2446 |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2447 if (!isIdentifierString(str)) |
365 | 2448 return false; |
2449 | |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2450 return (str in reserved_ids_table) !is null; |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
2451 } |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2452 |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2453 dchar decodeUTF8() |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2454 { |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2455 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8()."); |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2456 char* p = this.p; |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2457 dchar d = *p; |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2458 |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2459 ++p; // Move to second byte. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2460 // Error if second byte is not a trail byte. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2461 if (!isTrailByte(*p)) |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2462 goto Lerr2; |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2463 |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2464 // Check for overlong sequences. |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2465 switch (d) |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2466 { |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2467 case 0xE0, // 11100000 100xxxxx |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2468 0xF0, // 11110000 1000xxxx |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2469 0xF8, // 11111000 10000xxx |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2470 0xFC: // 11111100 100000xx |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2471 if ((*p & d) == 0x80) |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2472 goto Lerr; |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2473 default: |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2474 if ((d & 0xFE) == 0xC0) // 1100000x |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2475 goto Lerr; |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2476 } |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2477 |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2478 const char[] checkNextByte = "if (!isTrailByte(*++p))" |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2479 " goto Lerr2;"; |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2480 const char[] appendSixBits = "d = (d << 6) | *p & 0b0011_1111;"; |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2481 |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2482 // Decode |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2483 if ((d & 0b1110_0000) == 0b1100_0000) |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2484 { |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2485 // 110xxxxx 10xxxxxx |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2486 d &= 0b0001_1111; |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2487 mixin(appendSixBits); |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2488 } |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2489 else if ((d & 0b1111_0000) == 0b1110_0000) |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2490 { |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2491 // 1110xxxx 10xxxxxx 10xxxxxx |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2492 d &= 0b0000_1111; |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2493 mixin(appendSixBits ~ |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2494 checkNextByte ~ appendSixBits); |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2495 } |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2496 else if ((d & 0b1111_1000) == 0b1111_0000) |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2497 { |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2498 // 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2499 d &= 0b0000_0111; |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2500 mixin(appendSixBits ~ |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2501 checkNextByte ~ appendSixBits ~ |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2502 checkNextByte ~ appendSixBits); |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2503 } |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2504 else |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2505 // 5 and 6 byte UTF-8 sequences are not allowed yet. |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2506 // 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2507 // 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2508 goto Lerr; |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2509 |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2510 assert(isTrailByte(*p)); |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2511 |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2512 if (!isValidChar(d)) |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2513 { |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2514 Lerr: |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2515 // Three cases: |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2516 // *) the UTF-8 sequence was successfully decoded but the resulting |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2517 // character is invalid. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2518 // p points to last trail byte in the sequence. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2519 // *) the UTF-8 sequence is overlong. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2520 // p points to second byte in the sequence. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2521 // *) the UTF-8 sequence has more than 4 bytes or starts with |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2522 // a trail byte. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2523 // p points to second byte in the sequence. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2524 assert(isTrailByte(*p)); |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2525 // Move to next ASCII character or lead byte of a UTF-8 sequence. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2526 while (p < (end-1) && isTrailByte(*p)) |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2527 ++p; |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2528 --p; |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2529 assert(!isTrailByte(p[1])); |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2530 Lerr2: |
430
e6c759e151cd
Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
426
diff
changeset
|
2531 d = REPLACEMENT_CHAR; |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2532 error(this.p, MID.InvalidUTF8Sequence); |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2533 } |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2534 |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2535 this.p = p; |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2536 return d; |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2537 } |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
2538 |
519 | 2539 static void encodeUTF8(ref char[] str, dchar d) |
48 | 2540 { |
430
e6c759e151cd
Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
426
diff
changeset
|
2541 assert(!isascii(d), "check for ASCII char before calling encodeUTF8()."); |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2542 assert(isValidChar(d), "check if character is valid before calling encodeUTF8()."); |
430
e6c759e151cd
Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
426
diff
changeset
|
2543 |
519 | 2544 char[6] b = void; |
48 | 2545 if (d < 0x800) |
2546 { | |
2547 b[0] = 0xC0 | (d >> 6); | |
2548 b[1] = 0x80 | (d & 0x3F); | |
2549 str ~= b[0..2]; | |
2550 } | |
2551 else if (d < 0x10000) | |
2552 { | |
2553 b[0] = 0xE0 | (d >> 12); | |
2554 b[1] = 0x80 | ((d >> 6) & 0x3F); | |
2555 b[2] = 0x80 | (d & 0x3F); | |
2556 str ~= b[0..3]; | |
2557 } | |
2558 else if (d < 0x200000) | |
2559 { | |
2560 b[0] = 0xF0 | (d >> 18); | |
2561 b[1] = 0x80 | ((d >> 12) & 0x3F); | |
2562 b[2] = 0x80 | ((d >> 6) & 0x3F); | |
2563 b[3] = 0x80 | (d & 0x3F); | |
2564 str ~= b[0..4]; | |
2565 } | |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
2566 /+ // There are no 5 and 6 byte UTF-8 sequences yet. |
48 | 2567 else if (d < 0x4000000) |
2568 { | |
2569 b[0] = 0xF8 | (d >> 24); | |
2570 b[1] = 0x80 | ((d >> 18) & 0x3F); | |
2571 b[2] = 0x80 | ((d >> 12) & 0x3F); | |
2572 b[3] = 0x80 | ((d >> 6) & 0x3F); | |
2573 b[4] = 0x80 | (d & 0x3F); | |
2574 str ~= b[0..5]; | |
2575 } | |
2576 else if (d < 0x80000000) | |
2577 { | |
2578 b[0] = 0xFC | (d >> 30); | |
2579 b[1] = 0x80 | ((d >> 24) & 0x3F); | |
2580 b[2] = 0x80 | ((d >> 18) & 0x3F); | |
2581 b[3] = 0x80 | ((d >> 12) & 0x3F); | |
2582 b[4] = 0x80 | ((d >> 6) & 0x3F); | |
2583 b[5] = 0x80 | (d & 0x3F); | |
2584 str ~= b[0..6]; | |
2585 } | |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
2586 +/ |
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
2587 else |
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
2588 assert(0); |
48 | 2589 } |
5 | 2590 } |
39
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2591 |
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2592 unittest |
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2593 { |
391
33b566df6af4
Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
390
diff
changeset
|
2594 Stdout("Testing Lexer.\n"); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2595 struct Pair |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2596 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2597 char[] tokenText; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2598 TOK type; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2599 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2600 static Pair[] pairs = [ |
494
9a7ca8c56e59
Refactored a few things in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
490
diff
changeset
|
2601 {"#!äöüß", TOK.Shebang}, {"\n", TOK.Newline}, |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2602 {"//çay", TOK.Comment}, {"\n", TOK.Newline}, |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2603 {"&", TOK.AndBinary}, |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2604 {"/*çağ*/", TOK.Comment}, {"&&", TOK.AndLogical}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2605 {"/+çak+/", TOK.Comment}, {"&=", TOK.AndAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2606 {">", TOK.Greater}, {"+", TOK.Plus}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2607 {">=", TOK.GreaterEqual}, {"++", TOK.PlusPlus}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2608 {">>", TOK.RShift}, {"+=", TOK.PlusAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2609 {">>=", TOK.RShiftAssign}, {"-", TOK.Minus}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2610 {">>>", TOK.URShift}, {"--", TOK.MinusMinus}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2611 {">>>=", TOK.URShiftAssign}, {"-=", TOK.MinusAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2612 {"<", TOK.Less}, {"=", TOK.Assign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2613 {"<=", TOK.LessEqual}, {"==", TOK.Equal}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2614 {"<>", TOK.LorG}, {"~", TOK.Tilde}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2615 {"<>=", TOK.LorEorG}, {"~=", TOK.CatAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2616 {"<<", TOK.LShift}, {"*", TOK.Mul}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2617 {"<<=", TOK.LShiftAssign}, {"*=", TOK.MulAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2618 {"!", TOK.Not}, {"/", TOK.Div}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2619 {"!=", TOK.NotEqual}, {"/=", TOK.DivAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2620 {"!<", TOK.UorGorE}, {"^", TOK.Xor}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2621 {"!>", TOK.UorLorE}, {"^=", TOK.XorAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2622 {"!<=", TOK.UorG}, {"%", TOK.Mod}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2623 {"!>=", TOK.UorL}, {"%=", TOK.ModAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2624 {"!<>", TOK.UorE}, {"(", TOK.LParen}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2625 {"!<>=", TOK.Unordered}, {")", TOK.RParen}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2626 {".", TOK.Dot}, {"[", TOK.LBracket}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2627 {"..", TOK.Slice}, {"]", TOK.RBracket}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2628 {"...", TOK.Ellipses}, {"{", TOK.LBrace}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2629 {"|", TOK.OrBinary}, {"}", TOK.RBrace}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2630 {"||", TOK.OrLogical}, {":", TOK.Colon}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2631 {"|=", TOK.OrAssign}, {";", TOK.Semicolon}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2632 {"?", TOK.Question}, {",", TOK.Comma}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2633 {"$", TOK.Dollar}, {"cam", TOK.Identifier}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2634 {"çay", TOK.Identifier}, {".0", TOK.Float64}, |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2635 {"0", TOK.Int32}, {"\n", TOK.Newline}, |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2636 {"\r", TOK.Newline}, {"\r\n", TOK.Newline}, |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2637 {"\u2028", TOK.Newline}, {"\u2029", TOK.Newline} |
40 | 2638 ]; |
39
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2639 |
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2640 char[] src; |
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2641 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2642 // Join all token texts into a single string. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2643 foreach (i, pair; pairs) |
494
9a7ca8c56e59
Refactored a few things in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
490
diff
changeset
|
2644 if (pair.type == TOK.Comment && pair.tokenText[1] == '/' || // Line comment. |
9a7ca8c56e59
Refactored a few things in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
490
diff
changeset
|
2645 pair.type == TOK.Shebang) |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2646 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2647 assert(pairs[i+1].type == TOK.Newline); // Must be followed by a newline. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2648 src ~= pair.tokenText; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2649 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2650 else |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2651 src ~= pair.tokenText ~ " "; |
39
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2652 |
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2653 auto lx = new Lexer(src, ""); |
350 | 2654 auto token = lx.getTokens(); |
39
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2655 |
350 | 2656 uint i; |
2657 assert(token == lx.head); | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2658 assert(token.next.type == TOK.Newline); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2659 token = token.next.next; |
350 | 2660 do |
2661 { | |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2662 assert(i < pairs.length); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2663 assert(token.srcText == pairs[i].tokenText, Format("Scanned '{0}' but expected '{1}'", token.srcText, pairs[i].tokenText)); |
350 | 2664 ++i; |
2665 token = token.next; | |
2666 } while (token.type != TOK.EOF) | |
41 | 2667 } |
55 | 2668 |
67
996065105910
- Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents:
66
diff
changeset
|
2669 unittest |
996065105910
- Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents:
66
diff
changeset
|
2670 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2671 Stdout("Testing method Lexer.peek()\n"); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2672 string sourceText = "unittest { }"; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2673 auto lx = new Lexer(sourceText, null); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2674 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2675 auto next = lx.head; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2676 lx.peek(next); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2677 assert(next.type == TOK.Newline); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2678 lx.peek(next); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2679 assert(next.type == TOK.Unittest); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2680 lx.peek(next); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2681 assert(next.type == TOK.LBrace); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2682 lx.peek(next); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2683 assert(next.type == TOK.RBrace); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2684 lx.peek(next); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2685 assert(next.type == TOK.EOF); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2686 |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2687 lx = new Lexer("", null); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2688 next = lx.head; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2689 lx.peek(next); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2690 assert(next.type == TOK.Newline); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2691 lx.peek(next); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2692 assert(next.type == TOK.EOF); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2693 } |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2694 |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2695 unittest |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2696 { |
67
996065105910
- Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents:
66
diff
changeset
|
2697 // Numbers unittest |
68
7eb83dd38901
- Simplified suffix rule and added a few more numbers to unittest.
aziz
parents:
67
diff
changeset
|
2698 // 0L 0ULi 0_L 0_UL 0x0U 0x0p2 0_Fi 0_e2 0_F 0_i |
7eb83dd38901
- Simplified suffix rule and added a few more numbers to unittest.
aziz
parents:
67
diff
changeset
|
2699 // 0u 0U 0uL 0UL 0L 0LU 0Lu |
7eb83dd38901
- Simplified suffix rule and added a few more numbers to unittest.
aziz
parents:
67
diff
changeset
|
2700 // 0Li 0f 0F 0fi 0Fi 0i |
7eb83dd38901
- Simplified suffix rule and added a few more numbers to unittest.
aziz
parents:
67
diff
changeset
|
2701 // 0b_1_LU 0b1000u |
7eb83dd38901
- Simplified suffix rule and added a few more numbers to unittest.
aziz
parents:
67
diff
changeset
|
2702 // 0x232Lu |
67
996065105910
- Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents:
66
diff
changeset
|
2703 } |
996065105910
- Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents:
66
diff
changeset
|
2704 |
55 | 2705 /// ASCII character properties table. |
2706 static const int ptable[256] = [ | |
2707 0, 0, 0, 0, 0, 0, 0, 0, 0,32, 0,32,32, 0, 0, 0, | |
2708 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
2709 32, 0, 0x2200, 0, 0, 0, 0, 0x2700, 0, 0, 0, 0, 0, 0, 0, 0, | |
2710 7, 7, 7, 7, 7, 7, 7, 7, 6, 6, 0, 0, 0, 0, 0, 0x3f00, | |
2711 0,12,12,12,12,12,12, 8, 8, 8, 8, 8, 8, 8, 8, 8, | |
2712 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0, 0x5c00, 0, 0,16, | |
2713 0, 0x70c, 0x80c,12,12,12, 0xc0c, 8, 8, 8, 8, 8, 8, 8, 0xa08, 8, | |
2714 8, 8, 0xd08, 8, 0x908, 8, 0xb08, 8, 8, 8, 8, 0, 0, 0, 0, 0, | |
2715 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
2716 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
2717 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
2718 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
2719 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
2720 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
2721 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, | |
2722 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 | |
2723 ]; | |
2724 | |
2725 enum CProperty | |
2726 { | |
2727 Octal = 1, | |
2728 Digit = 1<<1, | |
2729 Hex = 1<<2, | |
2730 Alpha = 1<<3, | |
2731 Underscore = 1<<4, | |
2732 Whitespace = 1<<5 | |
2733 } | |
2734 | |
2735 const uint EVMask = 0xFF00; // Bit mask for escape value | |
2736 | |
2737 private alias CProperty CP; | |
2738 int isoctal(char c) { return ptable[c] & CP.Octal; } | |
2739 int isdigit(char c) { return ptable[c] & CP.Digit; } | |
2740 int ishexad(char c) { return ptable[c] & CP.Hex; } | |
2741 int isalpha(char c) { return ptable[c] & CP.Alpha; } | |
2742 int isalnum(char c) { return ptable[c] & (CP.Alpha | CP.Digit); } | |
2743 int isidbeg(char c) { return ptable[c] & (CP.Alpha | CP.Underscore); } | |
2744 int isident(char c) { return ptable[c] & (CP.Alpha | CP.Underscore | CP.Digit); } | |
2745 int isspace(char c) { return ptable[c] & CP.Whitespace; } | |
2746 int char2ev(char c) { return ptable[c] >> 8; /*(ptable[c] & EVMask) >> 8;*/ } | |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
2747 int isascii(uint c) { return c < 128; } |
55 | 2748 |
2749 version(gen_ptable) | |
2750 static this() | |
2751 { | |
2752 alias ptable p; | |
402
22d65b2bef4f
Migrated the code that generates the ptable to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
397
diff
changeset
|
2753 assert(p.length == 256); |
55 | 2754 // Initialize character properties table. |
2755 for (int i; i < p.length; ++i) | |
2756 { | |
402
22d65b2bef4f
Migrated the code that generates the ptable to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
397
diff
changeset
|
2757 p[i] = 0; // Reset |
55 | 2758 if ('0' <= i && i <= '7') |
2759 p[i] |= CP.Octal; | |
2760 if ('0' <= i && i <= '9') | |
2761 p[i] |= CP.Digit; | |
2762 if (isdigit(i) || 'a' <= i && i <= 'f' || 'A' <= i && i <= 'F') | |
2763 p[i] |= CP.Hex; | |
2764 if ('a' <= i && i <= 'z' || 'A' <= i && i <= 'Z') | |
2765 p[i] |= CP.Alpha; | |
2766 if (i == '_') | |
2767 p[i] |= CP.Underscore; | |
2768 if (i == ' ' || i == '\t' || i == '\v' || i == '\f') | |
2769 p[i] |= CP.Whitespace; | |
2770 } | |
2771 // Store escape sequence values in second byte. | |
2772 assert(CProperty.max <= ubyte.max, "character property flags and escape value byte overlap."); | |
2773 p['\''] |= 39 << 8; | |
2774 p['"'] |= 34 << 8; | |
2775 p['?'] |= 63 << 8; | |
2776 p['\\'] |= 92 << 8; | |
2777 p['a'] |= 7 << 8; | |
2778 p['b'] |= 8 << 8; | |
2779 p['f'] |= 12 << 8; | |
2780 p['n'] |= 10 << 8; | |
2781 p['r'] |= 13 << 8; | |
2782 p['t'] |= 9 << 8; | |
2783 p['v'] |= 11 << 8; | |
2784 // Print a formatted array literal. | |
2785 char[] array = "[\n"; | |
402
22d65b2bef4f
Migrated the code that generates the ptable to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
397
diff
changeset
|
2786 foreach (i, c; ptable) |
55 | 2787 { |
402
22d65b2bef4f
Migrated the code that generates the ptable to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
397
diff
changeset
|
2788 array ~= Format((c>255?" 0x{0:x},":"{0,2},"), c) ~ (((i+1) % 16) ? "":"\n"); |
55 | 2789 } |
2790 array[$-2..$] = "\n]"; | |
402
22d65b2bef4f
Migrated the code that generates the ptable to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
397
diff
changeset
|
2791 Stdout(array).newline; |
55 | 2792 } |