Mercurial > projects > dil
annotate trunk/src/dil/lexer/Lexer.d @ 762:c909a3d3fa52
Fixed vararg issue with gdc.
author | Aziz K?ksal <aziz.koeksal@gmail.com> |
---|---|
date | Sat, 16 Feb 2008 00:52:43 +0100 |
parents | 90668b83ae5e |
children | f26f13b5a3a3 |
rev | line source |
---|---|
0 | 1 /++ |
2 Author: Aziz Köksal | |
249 | 3 License: GPL3 |
0 | 4 +/ |
576
0df647660e76
Moved Lexer.d to new package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
552
diff
changeset
|
5 module dil.lexer.Lexer; |
577
9e811db780a6
Moved LexerFuncs.d to package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
576
diff
changeset
|
6 |
596
39fac5531b85
Moved dil.Token to dil.lexer.Token.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
577
diff
changeset
|
7 import dil.lexer.Token; |
599
c4cdea3a65ba
Moved dil.Keywords to dil.lexer.Keywords.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
596
diff
changeset
|
8 import dil.lexer.Keywords; |
600
041eae272362
Moved dil.Identifier to dil.lexer.Identifier.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
599
diff
changeset
|
9 import dil.lexer.Identifier; |
601
9f96fd51cb41
Moved dil.IdTable to dil.lexer.IdTable.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
600
diff
changeset
|
10 import dil.lexer.IdTable; |
327 | 11 import dil.Information; |
12 import dil.Messages; | |
13 import dil.HtmlEntities; | |
464
325714d8aa6c
Added new module with compiler constants to resolve import cycle.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
434
diff
changeset
|
14 import dil.CompilerInfo; |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
15 import dil.Unicode; |
755
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
16 import dil.SourceText; |
737
f88b5285b86b
Implemented DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
715
diff
changeset
|
17 import common; |
f88b5285b86b
Implemented DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
715
diff
changeset
|
18 |
391
33b566df6af4
Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
390
diff
changeset
|
19 import tango.stdc.stdlib : strtof, strtod, strtold; |
33b566df6af4
Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
390
diff
changeset
|
20 import tango.stdc.errno : errno, ERANGE; |
33b566df6af4
Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
390
diff
changeset
|
21 import tango.stdc.time : time_t, time, ctime; |
33b566df6af4
Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
390
diff
changeset
|
22 import tango.stdc.string : strlen; |
0 | 23 |
577
9e811db780a6
Moved LexerFuncs.d to package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
576
diff
changeset
|
24 public import dil.lexer.Funcs; |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
25 |
499
52447db67938
Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
498
diff
changeset
|
26 /++ |
52447db67938
Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
498
diff
changeset
|
27 The Lexer analyzes the characters of a source text and |
52447db67938
Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
498
diff
changeset
|
28 produces a doubly-linked list of tokens. |
52447db67938
Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
498
diff
changeset
|
29 +/ |
0 | 30 class Lexer |
31 { | |
755
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
32 SourceText srcText; /// The source text. |
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
33 char* p; /// Points to the current character in the source text. |
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
34 char* end; /// Points one character past the end of the source text. |
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
35 |
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
36 Token* head; /// The head of the doubly linked token list. |
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
37 Token* tail; /// The tail of the linked list. Set in scan(). |
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
38 Token* token; /// Points to the current token in the token list. |
4 | 39 |
434
cac9c8d637ad
Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
433
diff
changeset
|
40 // Members used for error messages: |
532
50e64bab9c7a
Renamed InformationManager to InfoManager.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
520
diff
changeset
|
41 InfoManager infoMan; |
514
6ddff941862a
Added new error classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
510
diff
changeset
|
42 LexerError[] errors; |
607
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
43 /// Always points to the first character of the current line. |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
44 char* lineBegin; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
45 // Token* newline; /// Current newline token. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
46 uint lineNum = 1; /// Current, actual source text line number. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
47 uint lineNum_hline; /// Line number set by #line. |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
48 uint inTokenString; /// > 0 if inside q{ } |
607
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
49 NewlineData.FilePaths* filePaths; |
103
511a1aa25896
- Added reportErrors member to Lexer. Moved peek() down and rewrote it a bit making use of reportErrors. error() uses reportErrors too.
aziz
parents:
94
diff
changeset
|
50 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
51 /++ |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
52 Construct a Lexer object. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
53 Params: |
755
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
54 srcText = the UTF-8 source code. |
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
55 infoMan = used for collecting error messages. |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
56 +/ |
755
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
57 this(SourceText srcText, InfoManager infoMan = null) |
4 | 58 { |
755
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
59 this.srcText = srcText; |
520
f203c5248d0b
Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
519
diff
changeset
|
60 this.infoMan = infoMan; |
30 | 61 |
755
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
62 assert(text.length && text[$-1] == 0, "source text has no sentinel character"); |
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
63 this.p = text.ptr; |
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
64 this.end = this.p + text.length; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
65 this.lineBegin = this.p; |
239
7911f6a92e6e
- Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents:
223
diff
changeset
|
66 |
7911f6a92e6e
- Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents:
223
diff
changeset
|
67 this.head = new Token; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
68 this.head.kind = TOK.HEAD; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
69 this.head.start = this.head.end = this.p; |
239
7911f6a92e6e
- Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents:
223
diff
changeset
|
70 this.token = this.head; |
607
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
71 // Initialize this.filePaths. |
755
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
72 newFilePath(this.srcText.filePath); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
73 // Add a newline as the first token after the head. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
74 auto newline = new Token; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
75 newline.kind = TOK.Newline; |
552
3bc7801c207e
Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
550
diff
changeset
|
76 newline.setWhitespaceFlag(); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
77 newline.start = newline.end = this.p; |
607
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
78 newline.newline.filePaths = this.filePaths; |
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
79 newline.newline.oriLineNum = 1; |
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
80 newline.newline.setLineNum = 0; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
81 // Link in. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
82 this.token.next = newline; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
83 newline.prev = this.token; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
84 this.token = newline; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
85 // this.newline = newline; |
315 | 86 scanShebang(); |
87 } | |
88 | |
344 | 89 ~this() |
90 { | |
91 auto token = head.next; | |
418
7354f15cd5e9
Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
416
diff
changeset
|
92 while (token !is null) |
344 | 93 { |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
94 assert(token.kind == TOK.EOF ? token == tail && token.next is null : 1); |
344 | 95 delete token.prev; |
96 token = token.next; | |
418
7354f15cd5e9
Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
416
diff
changeset
|
97 } |
344 | 98 delete tail; |
99 } | |
100 | |
755
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
101 char[] text() |
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
102 { |
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
103 return srcText.data; |
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
104 } |
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
105 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
106 /++ |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
107 The "shebang" may optionally appear once at the beginning of a file. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
108 Regexp: #![^\EndOfLine]* |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
109 +/ |
315 | 110 void scanShebang() |
111 { | |
112 if (*p == '#' && p[1] == '!') | |
113 { | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
114 auto t = new Token; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
115 t.kind = TOK.Shebang; |
552
3bc7801c207e
Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
550
diff
changeset
|
116 t.setWhitespaceFlag(); |
315 | 117 t.start = p; |
118 ++p; | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
119 while (!isEndOfLine(++p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
120 isascii(*p) || decodeUTF8(); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
121 t.end = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
122 this.token.next = t; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
123 t.prev = this.token; |
315 | 124 } |
4 | 125 } |
126 | |
343
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
127 void finalizeSpecialToken(ref Token t) |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
128 { |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
129 assert(t.srcText[0..2] == "__"); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
130 switch (t.kind) |
343
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
131 { |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
132 case TOK.FILE: |
607
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
133 t.str = this.filePaths.setPath; |
343
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
134 break; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
135 case TOK.LINE: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
136 t.uint_ = this.errorLineNumber(this.lineNum); |
343
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
137 break; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
138 case TOK.DATE, |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
139 TOK.TIME, |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
140 TOK.TIMESTAMP: |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
141 time_t time_val; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
142 time(&time_val); |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
143 char* str = ctime(&time_val); |
742
fc8f0e61bc42
Fixed building of time_str.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
737
diff
changeset
|
144 char[] time_str = str[0 .. strlen(str)-1]; // -1 removes trailing '\n'. |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
145 switch (t.kind) |
343
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
146 { |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
147 case TOK.DATE: |
346
ce9e2e77743f
- Fix: include one space when slicing time_str for __DATE__.
aziz
parents:
344
diff
changeset
|
148 time_str = time_str[4..11] ~ time_str[20..24] ~ \0; break; |
343
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
149 case TOK.TIME: |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
150 time_str = time_str[11..19] ~ \0; break; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
151 case TOK.TIMESTAMP: |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
152 time_str = time_str[0..24] ~ \0; break; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
153 default: assert(0); |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
154 } |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
155 t.str = time_str; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
156 break; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
157 case TOK.VENDOR: |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
158 t.str = VENDOR; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
159 break; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
160 case TOK.VERSION: |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
161 t.uint_ = VERSION_MAJOR*1000 + VERSION_MINOR; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
162 break; |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
163 default: |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
164 assert(0); |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
165 } |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
166 } |
95f1b6e43214
- Removed TOK.Special and added an own entry for each special token.
aziz
parents:
327
diff
changeset
|
167 |
607
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
168 /// Sets a new file path. |
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
169 void newFilePath(char[] newPath) |
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
170 { |
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
171 auto paths = new NewlineData.FilePaths; |
755
90668b83ae5e
Introduced new module dil.SourceText and class SourceText.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
742
diff
changeset
|
172 paths.oriPath = this.srcText.filePath; |
607
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
173 paths.setPath = newPath; |
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
174 this.filePaths = paths; |
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
175 } |
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
176 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
177 private void setLineBegin(char* p) |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
178 { |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
179 // Check that we can look behind one character. |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
180 assert((p-1) >= text.ptr && p < end); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
181 // Check that previous character is a newline. |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
182 assert(isNewlineEnd(p - 1)); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
183 this.lineBegin = p; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
184 } |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
185 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
186 private void scanNext(ref Token* t) |
419
89e40d43065d
Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
418
diff
changeset
|
187 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
188 assert(t !is null); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
189 if (t.next) |
434
cac9c8d637ad
Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
433
diff
changeset
|
190 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
191 t = t.next; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
192 // if (t.kind == TOK.Newline) |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
193 // this.newline = t; |
434
cac9c8d637ad
Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
433
diff
changeset
|
194 } |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
195 else if (t != this.tail) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
196 { |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
197 Token* new_t = new Token; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
198 scan(*new_t); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
199 new_t.prev = t; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
200 t.next = new_t; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
201 t = new_t; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
202 } |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
203 } |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
204 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
205 /// Advance t one token forward. |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
206 void peek(ref Token* t) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
207 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
208 scanNext(t); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
209 } |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
210 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
211 /// Advance to the next token in the source text. |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
212 TOK nextToken() |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
213 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
214 scanNext(this.token); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
215 return this.token.kind; |
419
89e40d43065d
Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
418
diff
changeset
|
216 } |
89e40d43065d
Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
418
diff
changeset
|
217 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
218 /// Returns true if p points to the last character of a Newline. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
219 bool isNewlineEnd(char* p) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
220 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
221 if (*p == '\n' || *p == '\r') |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
222 return true; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
223 if (*p == LS[2] || *p == PS[2]) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
224 if ((p-2) >= text.ptr) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
225 if (p[-1] == LS[1] && p[-2] == LS[0]) |
434
cac9c8d637ad
Fix: members related to error info weren't updated correctly.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
433
diff
changeset
|
226 return true; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
227 return false; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
228 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
229 |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
230 /++ |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
231 This is the old scan method. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
232 TODO: profile old and new to see which one is faster. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
233 +/ |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
234 public void scan(ref Token t) |
207 | 235 in |
3 | 236 { |
207 | 237 assert(text.ptr <= p && p < end); |
238 } | |
239 out | |
240 { | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
241 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind)); |
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
242 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind)); |
207 | 243 } |
244 body | |
245 { | |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
246 // Scan whitespace. |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
247 if (isspace(*p)) |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
248 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
249 t.ws = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
250 while (isspace(*++p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
251 {} |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
252 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
253 |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
254 // Scan a token. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
255 uint c = *p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
256 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
257 t.start = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
258 // Newline. |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
259 switch (*p) |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
260 { |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
261 case '\r': |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
262 if (p[1] == '\n') |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
263 ++p; |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
264 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
265 assert(isNewlineEnd(p)); |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
266 ++p; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
267 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
268 setLineBegin(p); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
269 // this.newline = &t; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
270 t.kind = TOK.Newline; |
552
3bc7801c207e
Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
550
diff
changeset
|
271 t.setWhitespaceFlag(); |
607
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
272 t.newline.filePaths = this.filePaths; |
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
273 t.newline.oriLineNum = lineNum; |
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
274 t.newline.setLineNum = lineNum_hline; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
275 t.end = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
276 return; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
277 default: |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
278 if (isUnicodeNewline(p)) |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
279 { |
412
fb31af0fda73
Added struct Location, and token2LocTable to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
411
diff
changeset
|
280 ++p; ++p; |
fb31af0fda73
Added struct Location, and token2LocTable to Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
411
diff
changeset
|
281 goto case '\n'; |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
282 } |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
283 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
284 // Identifier or string literal. |
13 | 285 if (isidbeg(c)) |
5 | 286 { |
33 | 287 if (c == 'r' && p[1] == '"' && ++p) |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
288 return scanRawStringLiteral(t); |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
289 if (c == 'x' && p[1] == '"') |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
290 return scanHexStringLiteral(t); |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
291 version(D2) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
292 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
293 if (c == 'q' && p[1] == '"') |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
294 return scanDelimitedStringLiteral(t); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
295 if (c == 'q' && p[1] == '{') |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
296 return scanTokenStringLiteral(t); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
297 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
298 // Scan identifier. |
12
0989206cf73c
- Added code to decode Unicode characters in identifiers.
aziz
parents:
11
diff
changeset
|
299 Lidentifier: |
5 | 300 do |
301 { c = *++p; } | |
550
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
302 while (isident(c) || !isascii(c) && isUnicodeAlpha()) |
28 | 303 |
5 | 304 t.end = p; |
28 | 305 |
502
4e14cd1b24da
Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
499
diff
changeset
|
306 auto id = IdTable.lookup(t.srcText); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
307 t.kind = id.type; |
502
4e14cd1b24da
Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
499
diff
changeset
|
308 t.ident = id; |
28 | 309 |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
310 if (t.kind == TOK.Identifier || t.isKeyword) |
411 | 311 return; |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
312 else if (t.isSpecialToken) |
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
313 finalizeSpecialToken(t); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
314 else if (t.kind == TOK.EOF) |
411 | 315 { |
316 tail = &t; | |
317 assert(t.srcText == "__EOF__"); | |
318 } | |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
319 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
320 assert(0, "unexpected token type: " ~ Token.toString(t.kind)); |
5 | 321 return; |
322 } | |
323 | |
15 | 324 if (isdigit(c)) |
325 return scanNumber(t); | |
326 | |
8 | 327 if (c == '/') |
5 | 328 { |
8 | 329 c = *++p; |
14 | 330 switch(c) |
5 | 331 { |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
332 case '=': |
14 | 333 ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
334 t.kind = TOK.DivAssign; |
14 | 335 t.end = p; |
336 return; | |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
337 case '+': |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
338 return scanNestedComment(t); |
32
d7011daa4740
- Added missing commas after the items in the messages table.
aziz
parents:
31
diff
changeset
|
339 case '*': |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
340 return scanBlockComment(t); |
32
d7011daa4740
- Added missing commas after the items in the messages table.
aziz
parents:
31
diff
changeset
|
341 case '/': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
342 while (!isEndOfLine(++p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
343 isascii(*p) || decodeUTF8(); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
344 t.kind = TOK.Comment; |
552
3bc7801c207e
Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
550
diff
changeset
|
345 t.setWhitespaceFlag(); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
346 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
347 return; |
32
d7011daa4740
- Added missing commas after the items in the messages table.
aziz
parents:
31
diff
changeset
|
348 default: |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
349 t.kind = TOK.Div; |
10 | 350 t.end = p; |
351 return; | |
352 } | |
5 | 353 } |
354 | |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
355 switch (c) |
20 | 356 { |
48 | 357 case '\'': |
358 return scanCharacterLiteral(t); | |
359 case '`': | |
360 return scanRawStringLiteral(t); | |
361 case '"': | |
362 return scanNormalStringLiteral(t); | |
363 case '\\': | |
364 char[] buffer; | |
365 do | |
366 { | |
367 c = scanEscapeSequence(); | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
368 if (isascii(c)) |
48 | 369 buffer ~= c; |
370 else | |
371 encodeUTF8(buffer, c); | |
372 } while (*p == '\\') | |
373 buffer ~= 0; | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
374 t.kind = TOK.String; |
48 | 375 t.str = buffer; |
376 t.end = p; | |
377 return; | |
38
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
378 case '>': /* > >= >> >>= >>> >>>= */ |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
379 c = *++p; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
380 switch (c) |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
381 { |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
382 case '=': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
383 t.kind = TOK.GreaterEqual; |
38
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
384 goto Lcommon; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
385 case '>': |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
386 if (p[1] == '>') |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
387 { |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
388 ++p; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
389 if (p[1] == '=') |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
390 { ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
391 t.kind = TOK.URShiftAssign; |
38
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
392 } |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
393 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
394 t.kind = TOK.URShift; |
38
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
395 } |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
396 else if (p[1] == '=') |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
397 { |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
398 ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
399 t.kind = TOK.RShiftAssign; |
38
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
400 } |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
401 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
402 t.kind = TOK.RShift; |
38
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
403 goto Lcommon; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
404 default: |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
405 t.kind = TOK.Greater; |
38
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
406 goto Lcommon2; |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
407 } |
640c45aaaaee
- Added code for parsing GreaterEqual, Greater, RShiftAssign, RShift, URShiftAssign and URShift tokens.
aziz
parents:
37
diff
changeset
|
408 assert(0); |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
409 case '<': /* < <= <> <>= << <<= */ |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
410 c = *++p; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
411 switch (c) |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
412 { |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
413 case '=': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
414 t.kind = TOK.LessEqual; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
415 goto Lcommon; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
416 case '<': |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
417 if (p[1] == '=') { |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
418 ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
419 t.kind = TOK.LShiftAssign; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
420 } |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
421 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
422 t.kind = TOK.LShift; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
423 goto Lcommon; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
424 case '>': |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
425 if (p[1] == '=') { |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
426 ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
427 t.kind = TOK.LorEorG; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
428 } |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
429 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
430 t.kind = TOK.LorG; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
431 goto Lcommon; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
432 default: |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
433 t.kind = TOK.Less; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
434 goto Lcommon2; |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
435 } |
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
436 assert(0); |
37
7f3bcb97d017
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
36
diff
changeset
|
437 case '!': /* ! !< !> !<= !>= !<> !<>= */ |
35
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
438 c = *++p; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
439 switch (c) |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
440 { |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
441 case '<': |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
442 c = *++p; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
443 if (c == '>') |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
444 { |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
445 if (p[1] == '=') { |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
446 ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
447 t.kind = TOK.Unordered; |
35
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
448 } |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
449 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
450 t.kind = TOK.UorE; |
35
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
451 } |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
452 else if (c == '=') |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
453 { |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
454 t.kind = TOK.UorG; |
35
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
455 } |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
456 else { |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
457 t.kind = TOK.UorGorE; |
35
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
458 goto Lcommon2; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
459 } |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
460 goto Lcommon; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
461 case '>': |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
462 if (p[1] == '=') |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
463 { |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
464 ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
465 t.kind = TOK.UorL; |
35
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
466 } |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
467 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
468 t.kind = TOK.UorLorE; |
35
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
469 goto Lcommon; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
470 case '=': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
471 t.kind = TOK.NotEqual; |
35
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
472 goto Lcommon; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
473 default: |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
474 t.kind = TOK.Not; |
35
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
475 goto Lcommon2; |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
476 } |
c470b9356e35
- Added code for parsing Unordered, UorE, UorG, UorGorE, UorL, UorLorE, NotEqual and Not tokens.
aziz
parents:
34
diff
changeset
|
477 assert(0); |
63 | 478 case '.': /* . .[0-9] .. ... */ |
22
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
479 if (p[1] == '.') |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
480 { |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
481 ++p; |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
482 if (p[1] == '.') { |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
483 ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
484 t.kind = TOK.Ellipses; |
22
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
485 } |
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
486 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
487 t.kind = TOK.Slice; |
22
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
488 } |
63 | 489 else if (isdigit(p[1])) |
490 { | |
491 return scanReal(t); | |
492 } | |
22
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
493 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
494 t.kind = TOK.Dot; |
22
b05fff8e2ce4
- Added code for parsing Dot, Slice and Ellipses tokens.
aziz
parents:
21
diff
changeset
|
495 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
496 case '|': /* | || |= */ |
23
1a7903701a3d
- Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents:
22
diff
changeset
|
497 c = *++p; |
1a7903701a3d
- Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents:
22
diff
changeset
|
498 if (c == '=') |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
499 t.kind = TOK.OrAssign; |
23
1a7903701a3d
- Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents:
22
diff
changeset
|
500 else if (c == '|') |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
501 t.kind = TOK.OrLogical; |
26 | 502 else { |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
503 t.kind = TOK.OrBinary; |
26 | 504 goto Lcommon2; |
505 } | |
23
1a7903701a3d
- Added code for parsing OrAssign, OrLogical and OrBinary tokens.
aziz
parents:
22
diff
changeset
|
506 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
507 case '&': /* & && &= */ |
24
903f91163f23
- Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents:
23
diff
changeset
|
508 c = *++p; |
903f91163f23
- Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents:
23
diff
changeset
|
509 if (c == '=') |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
510 t.kind = TOK.AndAssign; |
24
903f91163f23
- Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents:
23
diff
changeset
|
511 else if (c == '&') |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
512 t.kind = TOK.AndLogical; |
26 | 513 else { |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
514 t.kind = TOK.AndBinary; |
26 | 515 goto Lcommon2; |
516 } | |
24
903f91163f23
- Added code for parsing AndAssign, AndLogical and AndBinary tokens.
aziz
parents:
23
diff
changeset
|
517 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
518 case '+': /* + ++ += */ |
25
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
519 c = *++p; |
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
520 if (c == '=') |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
521 t.kind = TOK.PlusAssign; |
25
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
522 else if (c == '+') |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
523 t.kind = TOK.PlusPlus; |
26 | 524 else { |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
525 t.kind = TOK.Plus; |
26 | 526 goto Lcommon2; |
527 } | |
25
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
528 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
529 case '-': /* - -- -= */ |
25
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
530 c = *++p; |
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
531 if (c == '=') |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
532 t.kind = TOK.MinusAssign; |
25
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
533 else if (c == '-') |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
534 t.kind = TOK.MinusMinus; |
26 | 535 else { |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
536 t.kind = TOK.Minus; |
26 | 537 goto Lcommon2; |
538 } | |
539 goto Lcommon; | |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
540 case '=': /* = == */ |
26 | 541 if (p[1] == '=') { |
542 ++p; | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
543 t.kind = TOK.Equal; |
26 | 544 } |
25
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
545 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
546 t.kind = TOK.Assign; |
25
9c866208b3f6
- Added code for parsing PlusAssign, PlusPlus, Plus, MinusAssign, MinusMinus, Minus tokens.
aziz
parents:
24
diff
changeset
|
547 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
548 case '~': /* ~ ~= */ |
27 | 549 if (p[1] == '=') { |
550 ++p; | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
551 t.kind = TOK.CatAssign; |
27 | 552 } |
553 else | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
554 t.kind = TOK.Tilde; |
27 | 555 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
556 case '*': /* * *= */ |
29
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
557 if (p[1] == '=') { |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
558 ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
559 t.kind = TOK.MulAssign; |
29
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
560 } |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
561 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
562 t.kind = TOK.Mul; |
29
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
563 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
564 case '^': /* ^ ^= */ |
29
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
565 if (p[1] == '=') { |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
566 ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
567 t.kind = TOK.XorAssign; |
29
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
568 } |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
569 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
570 t.kind = TOK.Xor; |
29
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
571 goto Lcommon; |
36
3c7210a722f7
- Added code for parsing LorEorG, LorG, LessEqual, LessThan, LShiftAssign and LShift tokens.
aziz
parents:
35
diff
changeset
|
572 case '%': /* % %= */ |
29
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
573 if (p[1] == '=') { |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
574 ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
575 t.kind = TOK.ModAssign; |
29
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
576 } |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
577 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
578 t.kind = TOK.Mod; |
29
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
579 goto Lcommon; |
ef83eea26bbd
- Added code for parsing MulAssign, Mul, ModAssign, Mod, XorAssign and Xor tokens.
aziz
parents:
28
diff
changeset
|
580 // Single character tokens: |
20 | 581 case '(': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
582 t.kind = TOK.LParen; |
20 | 583 goto Lcommon; |
584 case ')': | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
585 t.kind = TOK.RParen; |
20 | 586 goto Lcommon; |
587 case '[': | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
588 t.kind = TOK.LBracket; |
20 | 589 goto Lcommon; |
590 case ']': | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
591 t.kind = TOK.RBracket; |
20 | 592 goto Lcommon; |
593 case '{': | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
594 t.kind = TOK.LBrace; |
20 | 595 goto Lcommon; |
596 case '}': | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
597 t.kind = TOK.RBrace; |
21
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
598 goto Lcommon; |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
599 case ':': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
600 t.kind = TOK.Colon; |
21
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
601 goto Lcommon; |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
602 case ';': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
603 t.kind = TOK.Semicolon; |
21
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
604 goto Lcommon; |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
605 case '?': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
606 t.kind = TOK.Question; |
21
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
607 goto Lcommon; |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
608 case ',': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
609 t.kind = TOK.Comma; |
21
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
610 goto Lcommon; |
c785c122e4e6
- Added code for parsing Colon, Semicolon, Question, Comma and Dollar.
aziz
parents:
20
diff
changeset
|
611 case '$': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
612 t.kind = TOK.Dollar; |
20 | 613 Lcommon: |
614 ++p; | |
26 | 615 Lcommon2: |
20 | 616 t.end = p; |
617 return; | |
30 | 618 case '#': |
360
b6a3755eba94
- Renamed scanSpecialToken() to scanSpecialTokenSequence().
aziz
parents:
350
diff
changeset
|
619 return scanSpecialTokenSequence(t); |
20 | 620 default: |
621 } | |
622 | |
411 | 623 // Check for EOF |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
624 if (isEOF(c)) |
411 | 625 { |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
626 assert(isEOF(*p), ""~*p); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
627 t.kind = TOK.EOF; |
411 | 628 t.end = p; |
629 tail = &t; | |
630 assert(t.start == t.end); | |
631 return; | |
632 } | |
633 | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
634 if (!isascii(c)) |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
635 { |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
636 c = decodeUTF8(); |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
637 if (isUniAlpha(c)) |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
638 goto Lidentifier; |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
639 } |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
640 |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
641 error(t.start, MID.IllegalCharacter, cast(dchar)c); |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
642 |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
643 ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
644 t.kind = TOK.Illegal; |
552
3bc7801c207e
Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
550
diff
changeset
|
645 t.setWhitespaceFlag(); |
390
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
646 t.dchar_ = c; |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
647 t.end = p; |
4d36eea1bbc9
Refactored Lexer.scan().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
389
diff
changeset
|
648 return; |
4 | 649 } |
3 | 650 } |
4 | 651 |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
652 template toUint(char[] T) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
653 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
654 static assert(0 < T.length && T.length <= 4); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
655 static if (T.length == 1) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
656 const uint toUint = T[0]; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
657 else |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
658 const uint toUint = (T[0] << ((T.length-1)*8)) | toUint!(T[1..$]); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
659 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
660 static assert(toUint!("\xAA\xBB\xCC\xDD") == 0xAABBCCDD); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
661 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
662 // Can't use this yet due to a bug in DMD (bug id=1534). |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
663 template case_(char[] str, TOK tok, char[] label) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
664 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
665 const char[] case_ = |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
666 `case `~toUint!(str).stringof~`: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
667 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
668 goto `~label~`;`; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
669 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
670 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
671 template case_L4(char[] str, TOK tok) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
672 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
673 const char[] case_L4 = case_!(str, tok, "Lcommon_4"); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
674 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
675 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
676 template case_L3(char[] str, TOK tok) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
677 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
678 const char[] case_L3 = case_!(str, tok, "Lcommon_3"); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
679 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
680 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
681 template case_L2(char[] str, TOK tok) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
682 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
683 const char[] case_L2 = case_!(str, tok, "Lcommon_2"); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
684 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
685 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
686 template case_L1(char[] str, TOK tok) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
687 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
688 const char[] case_L3 = case_!(str, tok, "Lcommon"); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
689 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
690 |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
691 public void scan_(ref Token t) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
692 in |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
693 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
694 assert(text.ptr <= p && p < end); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
695 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
696 out |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
697 { |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
698 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind)); |
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
699 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind)); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
700 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
701 body |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
702 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
703 // Scan whitespace. |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
704 if (isspace(*p)) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
705 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
706 t.ws = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
707 while (isspace(*++p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
708 {} |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
709 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
710 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
711 // Scan a token. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
712 t.start = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
713 // Newline. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
714 switch (*p) |
418
7354f15cd5e9
Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
416
diff
changeset
|
715 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
716 case '\r': |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
717 if (p[1] == '\n') |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
718 ++p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
719 case '\n': |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
720 assert(isNewlineEnd(p)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
721 ++p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
722 ++lineNum; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
723 setLineBegin(p); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
724 // this.newline = &t; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
725 t.kind = TOK.Newline; |
552
3bc7801c207e
Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
550
diff
changeset
|
726 t.setWhitespaceFlag(); |
607
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
727 t.newline.filePaths = this.filePaths; |
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
728 t.newline.oriLineNum = lineNum; |
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
729 t.newline.setLineNum = lineNum_hline; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
730 t.end = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
731 return; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
732 default: |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
733 if (isUnicodeNewline(p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
734 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
735 ++p; ++p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
736 goto case '\n'; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
737 } |
418
7354f15cd5e9
Applied some fixes to the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
416
diff
changeset
|
738 } |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
739 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
740 uint c = *p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
741 assert(end - p != 0); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
742 switch (end - p) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
743 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
744 case 1: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
745 goto L1character; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
746 case 2: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
747 c <<= 8; c |= p[1]; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
748 goto L2characters; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
749 case 3: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
750 c <<= 8; c |= p[1]; c <<= 8; c |= p[2]; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
751 goto L3characters; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
752 default: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
753 version(BigEndian) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
754 c = *cast(uint*)p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
755 else |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
756 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
757 c <<= 8; c |= p[1]; c <<= 8; c |= p[2]; c <<= 8; c |= p[3]; |
414
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
758 /+ |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
759 c = *cast(uint*)p; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
760 asm |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
761 { |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
762 mov EDX, c; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
763 bswap EDX; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
764 mov c, EDX; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
765 } |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
766 +/ |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
767 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
768 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
769 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
770 // 4 character tokens. |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
771 switch (c) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
772 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
773 case toUint!(">>>="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
774 t.kind = TOK.RShiftAssign; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
775 goto Lcommon_4; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
776 case toUint!("!<>="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
777 t.kind = TOK.Unordered; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
778 Lcommon_4: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
779 p += 4; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
780 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
781 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
782 default: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
783 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
784 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
785 c >>>= 8; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
786 L3characters: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
787 assert(p == t.start); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
788 // 3 character tokens. |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
789 switch (c) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
790 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
791 case toUint!(">>="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
792 t.kind = TOK.RShiftAssign; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
793 goto Lcommon_3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
794 case toUint!(">>>"): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
795 t.kind = TOK.URShift; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
796 goto Lcommon_3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
797 case toUint!("<>="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
798 t.kind = TOK.LorEorG; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
799 goto Lcommon_3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
800 case toUint!("<<="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
801 t.kind = TOK.LShiftAssign; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
802 goto Lcommon_3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
803 case toUint!("!<="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
804 t.kind = TOK.UorG; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
805 goto Lcommon_3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
806 case toUint!("!>="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
807 t.kind = TOK.UorL; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
808 goto Lcommon_3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
809 case toUint!("!<>"): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
810 t.kind = TOK.UorE; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
811 goto Lcommon_3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
812 case toUint!("..."): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
813 t.kind = TOK.Ellipses; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
814 Lcommon_3: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
815 p += 3; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
816 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
817 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
818 default: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
819 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
820 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
821 c >>>= 8; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
822 L2characters: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
823 assert(p == t.start); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
824 // 2 character tokens. |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
825 switch (c) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
826 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
827 case toUint!("/+"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
828 ++p; // Skip / |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
829 return scanNestedComment(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
830 case toUint!("/*"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
831 ++p; // Skip / |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
832 return scanBlockComment(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
833 case toUint!("//"): |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
834 ++p; // Skip / |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
835 assert(*p == '/'); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
836 while (!isEndOfLine(++p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
837 isascii(*p) || decodeUTF8(); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
838 t.kind = TOK.Comment; |
552
3bc7801c207e
Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
550
diff
changeset
|
839 t.setWhitespaceFlag(); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
840 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
841 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
842 case toUint!(">="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
843 t.kind = TOK.GreaterEqual; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
844 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
845 case toUint!(">>"): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
846 t.kind = TOK.RShift; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
847 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
848 case toUint!("<<"): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
849 t.kind = TOK.LShift; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
850 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
851 case toUint!("<="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
852 t.kind = TOK.LessEqual; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
853 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
854 case toUint!("<>"): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
855 t.kind = TOK.LorG; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
856 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
857 case toUint!("!<"): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
858 t.kind = TOK.UorGorE; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
859 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
860 case toUint!("!>"): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
861 t.kind = TOK.UorLorE; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
862 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
863 case toUint!("!="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
864 t.kind = TOK.NotEqual; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
865 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
866 case toUint!(".."): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
867 t.kind = TOK.Slice; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
868 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
869 case toUint!("&&"): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
870 t.kind = TOK.AndLogical; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
871 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
872 case toUint!("&="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
873 t.kind = TOK.AndAssign; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
874 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
875 case toUint!("||"): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
876 t.kind = TOK.OrLogical; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
877 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
878 case toUint!("|="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
879 t.kind = TOK.OrAssign; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
880 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
881 case toUint!("++"): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
882 t.kind = TOK.PlusPlus; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
883 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
884 case toUint!("+="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
885 t.kind = TOK.PlusAssign; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
886 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
887 case toUint!("--"): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
888 t.kind = TOK.MinusMinus; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
889 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
890 case toUint!("-="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
891 t.kind = TOK.MinusAssign; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
892 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
893 case toUint!("=="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
894 t.kind = TOK.Equal; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
895 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
896 case toUint!("~="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
897 t.kind = TOK.CatAssign; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
898 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
899 case toUint!("*="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
900 t.kind = TOK.MulAssign; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
901 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
902 case toUint!("/="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
903 t.kind = TOK.DivAssign; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
904 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
905 case toUint!("^="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
906 t.kind = TOK.XorAssign; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
907 goto Lcommon_2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
908 case toUint!("%="): |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
909 t.kind = TOK.ModAssign; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
910 Lcommon_2: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
911 p += 2; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
912 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
913 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
914 default: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
915 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
916 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
917 c >>>= 8; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
918 L1character: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
919 assert(p == t.start); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
920 assert(*p == c, Format("p={0},c={1}", *p, cast(dchar)c)); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
921 // 1 character tokens. |
414
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
922 // TODO: consider storing the token type in ptable. |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
923 switch (c) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
924 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
925 case '\'': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
926 return scanCharacterLiteral(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
927 case '`': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
928 return scanRawStringLiteral(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
929 case '"': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
930 return scanNormalStringLiteral(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
931 case '\\': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
932 char[] buffer; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
933 do |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
934 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
935 c = scanEscapeSequence(); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
936 if (isascii(c)) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
937 buffer ~= c; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
938 else |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
939 encodeUTF8(buffer, c); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
940 } while (*p == '\\') |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
941 buffer ~= 0; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
942 t.kind = TOK.String; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
943 t.str = buffer; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
944 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
945 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
946 case '<': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
947 t.kind = TOK.Greater; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
948 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
949 case '>': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
950 t.kind = TOK.Less; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
951 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
952 case '^': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
953 t.kind = TOK.Xor; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
954 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
955 case '!': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
956 t.kind = TOK.Not; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
957 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
958 case '.': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
959 if (isdigit(p[1])) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
960 return scanReal(t); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
961 t.kind = TOK.Dot; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
962 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
963 case '&': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
964 t.kind = TOK.AndBinary; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
965 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
966 case '|': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
967 t.kind = TOK.OrBinary; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
968 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
969 case '+': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
970 t.kind = TOK.Plus; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
971 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
972 case '-': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
973 t.kind = TOK.Minus; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
974 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
975 case '=': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
976 t.kind = TOK.Assign; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
977 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
978 case '~': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
979 t.kind = TOK.Tilde; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
980 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
981 case '*': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
982 t.kind = TOK.Mul; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
983 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
984 case '/': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
985 t.kind = TOK.Div; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
986 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
987 case '%': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
988 t.kind = TOK.Mod; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
989 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
990 case '(': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
991 t.kind = TOK.LParen; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
992 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
993 case ')': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
994 t.kind = TOK.RParen; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
995 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
996 case '[': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
997 t.kind = TOK.LBracket; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
998 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
999 case ']': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1000 t.kind = TOK.RBracket; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1001 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1002 case '{': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1003 t.kind = TOK.LBrace; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1004 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1005 case '}': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1006 t.kind = TOK.RBrace; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1007 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1008 case ':': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1009 t.kind = TOK.Colon; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1010 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1011 case ';': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1012 t.kind = TOK.Semicolon; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1013 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1014 case '?': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1015 t.kind = TOK.Question; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1016 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1017 case ',': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1018 t.kind = TOK.Comma; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1019 goto Lcommon; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1020 case '$': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1021 t.kind = TOK.Dollar; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1022 Lcommon: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1023 ++p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1024 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1025 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1026 case '#': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1027 return scanSpecialTokenSequence(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1028 default: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1029 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1030 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1031 assert(p == t.start); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1032 assert(*p == c); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1033 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1034 // TODO: consider moving isidbeg() and isdigit() up. |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1035 if (isidbeg(c)) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1036 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1037 if (c == 'r' && p[1] == '"' && ++p) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1038 return scanRawStringLiteral(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1039 if (c == 'x' && p[1] == '"') |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1040 return scanHexStringLiteral(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1041 version(D2) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1042 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1043 if (c == 'q' && p[1] == '"') |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1044 return scanDelimitedStringLiteral(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1045 if (c == 'q' && p[1] == '{') |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1046 return scanTokenStringLiteral(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1047 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1048 // Scan identifier. |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1049 Lidentifier: |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1050 do |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1051 { c = *++p; } |
550
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
1052 while (isident(c) || !isascii(c) && isUnicodeAlpha()) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1053 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1054 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1055 |
502
4e14cd1b24da
Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
499
diff
changeset
|
1056 auto id = IdTable.lookup(t.srcText); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1057 t.kind = id.type; |
502
4e14cd1b24da
Refactored code and added modules related to tabulated Identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
499
diff
changeset
|
1058 t.ident = id; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1059 |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1060 if (t.kind == TOK.Identifier || t.isKeyword) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1061 return; |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
1062 else if (t.isSpecialToken) |
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
1063 finalizeSpecialToken(t); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1064 else if (t.kind == TOK.EOF) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1065 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1066 tail = &t; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1067 assert(t.srcText == "__EOF__"); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1068 } |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
1069 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1070 assert(0, "unexpected token type: " ~ Token.toString(t.kind)); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1071 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1072 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1073 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1074 if (isdigit(c)) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1075 return scanNumber(t); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1076 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1077 // Check for EOF |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
1078 if (isEOF(c)) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1079 { |
497
0ffcc4ff82f3
Refactored a few things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
496
diff
changeset
|
1080 assert(isEOF(*p), *p~""); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1081 t.kind = TOK.EOF; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1082 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1083 tail = &t; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1084 assert(t.start == t.end); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1085 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1086 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1087 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1088 if (!isascii(c)) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1089 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1090 c = decodeUTF8(); |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1091 if (isUniAlpha(c)) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1092 goto Lidentifier; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1093 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1094 |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1095 error(t.start, MID.IllegalCharacter, cast(dchar)c); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1096 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1097 ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1098 t.kind = TOK.Illegal; |
552
3bc7801c207e
Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
550
diff
changeset
|
1099 t.setWhitespaceFlag(); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1100 t.dchar_ = c; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1101 t.end = p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1102 return; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1103 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1104 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1105 void scanBlockComment(ref Token t) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1106 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1107 assert(p[-1] == '/' && *p == '*'); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1108 auto tokenLineNum = lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1109 auto tokenLineBegin = lineBegin; |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1110 Loop: |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1111 while (1) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1112 { |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1113 switch (*++p) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1114 { |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1115 case '*': |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1116 if (p[1] != '/') |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1117 continue; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1118 p += 2; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1119 break Loop; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1120 case '\r': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1121 if (p[1] == '\n') |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1122 ++p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1123 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1124 assert(isNewlineEnd(p)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1125 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1126 setLineBegin(p+1); |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1127 break; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1128 default: |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1129 if (!isascii(*p)) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1130 { |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1131 if (isUnicodeNewlineChar(decodeUTF8())) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1132 goto case '\n'; |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1133 } |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1134 else if (isEOF(*p)) |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1135 { |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1136 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedBlockComment); |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1137 break Loop; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1138 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1139 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1140 } |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1141 t.kind = TOK.Comment; |
552
3bc7801c207e
Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
550
diff
changeset
|
1142 t.setWhitespaceFlag(); |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1143 t.end = p; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1144 return; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1145 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1146 |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1147 void scanNestedComment(ref Token t) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1148 { |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1149 assert(p[-1] == '/' && *p == '+'); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1150 auto tokenLineNum = lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1151 auto tokenLineBegin = lineBegin; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1152 uint level = 1; |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1153 Loop: |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1154 while (1) |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1155 { |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1156 switch (*++p) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1157 { |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1158 case '/': |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1159 if (p[1] == '+') |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1160 ++p, ++level; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1161 continue; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1162 case '+': |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1163 if (p[1] != '/') |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1164 continue; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1165 ++p; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1166 if (--level != 0) |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1167 continue; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1168 ++p; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1169 break Loop; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1170 case '\r': |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1171 if (p[1] == '\n') |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1172 ++p; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1173 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1174 assert(isNewlineEnd(p)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1175 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1176 setLineBegin(p+1); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1177 continue; |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1178 default: |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1179 if (!isascii(*p)) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1180 { |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1181 if (isUnicodeNewlineChar(decodeUTF8())) |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1182 goto case '\n'; |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1183 } |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1184 else if (isEOF(*p)) |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1185 { |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1186 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedNestedComment); |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1187 break Loop; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1188 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1189 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1190 } |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1191 t.kind = TOK.Comment; |
552
3bc7801c207e
Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
550
diff
changeset
|
1192 t.setWhitespaceFlag(); |
498
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1193 t.end = p; |
49c201b5c465
Refactored scanners for block and nested comments.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
497
diff
changeset
|
1194 return; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1195 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
1196 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1197 char scanPostfix() |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1198 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1199 assert(p[-1] == '"' || p[-1] == '`' || |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1200 { version(D2) return p[-1] == '}'; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1201 else return 0; }() |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1202 ); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1203 switch (*p) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1204 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1205 case 'c': |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1206 case 'w': |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1207 case 'd': |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1208 return *p++; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1209 default: |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1210 return 0; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1211 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1212 assert(0); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1213 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1214 |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1215 void scanNormalStringLiteral(ref Token t) |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1216 { |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1217 assert(*p == '"'); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1218 auto tokenLineNum = lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1219 auto tokenLineBegin = lineBegin; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1220 t.kind = TOK.String; |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1221 char[] buffer; |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1222 uint c; |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1223 while (1) |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1224 { |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1225 c = *++p; |
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1226 switch (c) |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1227 { |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1228 case '"': |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1229 ++p; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1230 t.pf = scanPostfix(); |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1231 Lreturn: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1232 t.str = buffer ~ '\0'; |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1233 t.end = p; |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1234 return; |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1235 case '\\': |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1236 c = scanEscapeSequence(); |
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1237 --p; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1238 if (isascii(c)) |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1239 break; |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
1240 encodeUTF8(buffer, c); |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1241 continue; |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1242 case '\r': |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1243 if (p[1] == '\n') |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1244 ++p; |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1245 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1246 assert(isNewlineEnd(p)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1247 c = '\n'; // Convert Newline to \n. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1248 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1249 setLineBegin(p+1); |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1250 break; |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1251 case 0, _Z_: |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1252 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedString); |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1253 goto Lreturn; |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1254 default: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1255 if (!isascii(c)) |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1256 { |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1257 c = decodeUTF8(); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1258 if (isUnicodeNewlineChar(c)) |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1259 goto case '\n'; |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1260 encodeUTF8(buffer, c); |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1261 continue; |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1262 } |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1263 } |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
1264 assert(isascii(c)); |
384
f7ce725e79c3
Refactored scanNormalStringLiteral().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
383
diff
changeset
|
1265 buffer ~= c; |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1266 } |
67
996065105910
- Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents:
66
diff
changeset
|
1267 assert(0); |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1268 } |
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
1269 |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1270 void scanCharacterLiteral(ref Token t) |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1271 { |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1272 assert(*p == '\''); |
46 | 1273 ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1274 t.kind = TOK.CharLiteral; |
46 | 1275 switch (*p) |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1276 { |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1277 case '\\': |
46 | 1278 t.dchar_ = scanEscapeSequence(); |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1279 break; |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1280 case '\'': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1281 error(t.start, MID.EmptyCharacterLiteral); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1282 break; |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1283 default: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1284 if (isEndOfLine(p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1285 break; |
46 | 1286 uint c = *p; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1287 if (!isascii(c)) |
49 | 1288 c = decodeUTF8(); |
46 | 1289 t.dchar_ = c; |
1290 ++p; | |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1291 } |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1292 |
46 | 1293 if (*p == '\'') |
1294 ++p; | |
1295 else | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1296 error(t.start, MID.UnterminatedCharacterLiteral); |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1297 t.end = p; |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1298 } |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
1299 |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1300 void scanRawStringLiteral(ref Token t) |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1301 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1302 assert(*p == '`' || *p == '"' && p[-1] == 'r'); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1303 auto tokenLineNum = lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1304 auto tokenLineBegin = lineBegin; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1305 t.kind = TOK.String; |
33 | 1306 uint delim = *p; |
1307 char[] buffer; | |
1308 uint c; | |
1309 while (1) | |
1310 { | |
1311 c = *++p; | |
1312 switch (c) | |
1313 { | |
1314 case '\r': | |
1315 if (p[1] == '\n') | |
1316 ++p; | |
1317 case '\n': | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1318 assert(isNewlineEnd(p)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1319 c = '\n'; // Convert Newline to '\n'. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1320 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1321 setLineBegin(p+1); |
52
f65a83c27638
- Fixed the raw string literal scanner. Newlines weren't copied to the buffer. Converting LS and PS to '\n' as well.
aziz
parents:
51
diff
changeset
|
1322 break; |
33 | 1323 case '`': |
1324 case '"': | |
1325 if (c == delim) | |
1326 { | |
1327 ++p; | |
1328 t.pf = scanPostfix(); | |
1329 Lreturn: | |
1330 t.str = buffer ~ '\0'; | |
1331 t.end = p; | |
1332 return; | |
1333 } | |
1334 break; | |
1335 case 0, _Z_: | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1336 error(tokenLineNum, tokenLineBegin, t.start, |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1337 delim == 'r' ? MID.UnterminatedRawString : MID.UnterminatedBackQuoteString); |
33 | 1338 goto Lreturn; |
1339 default: | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1340 if (!isascii(c)) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1341 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1342 c = decodeUTF8(); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1343 if (isUnicodeNewlineChar(c)) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1344 goto case '\n'; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1345 encodeUTF8(buffer, c); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1346 continue; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1347 } |
33 | 1348 } |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
1349 assert(isascii(c)); |
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
1350 buffer ~= c; |
33 | 1351 } |
1352 assert(0); | |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1353 } |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1354 |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1355 void scanHexStringLiteral(ref Token t) |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1356 { |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1357 assert(p[0] == 'x' && p[1] == '"'); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1358 t.kind = TOK.String; |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1359 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1360 auto tokenLineNum = lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1361 auto tokenLineBegin = lineBegin; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1362 |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1363 uint c; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1364 ubyte[] buffer; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1365 ubyte h; // hex number |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1366 uint n; // number of hex digits |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1367 |
53
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1368 ++p; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1369 assert(*p == '"'); |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1370 while (1) |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1371 { |
53
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1372 c = *++p; |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1373 switch (c) |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1374 { |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1375 case '"': |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1376 if (n & 1) |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1377 error(tokenLineNum, tokenLineBegin, t.start, MID.OddNumberOfDigitsInHexString); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1378 ++p; |
53
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1379 t.pf = scanPostfix(); |
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1380 Lreturn: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1381 t.str = cast(string) (buffer ~= 0); |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1382 t.end = p; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1383 return; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1384 case '\r': |
53
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1385 if (p[1] == '\n') |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1386 ++p; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1387 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1388 assert(isNewlineEnd(p)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1389 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1390 setLineBegin(p+1); |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1391 continue; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1392 default: |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1393 if (ishexad(c)) |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1394 { |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1395 if (c <= '9') |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1396 c -= '0'; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1397 else if (c <= 'F') |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1398 c -= 'A' - 10; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1399 else |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1400 c -= 'a' - 10; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1401 |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1402 if (n & 1) |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1403 { |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1404 h <<= 4; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1405 h |= c; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1406 buffer ~= h; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1407 } |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1408 else |
112 | 1409 h = cast(ubyte)c; |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1410 ++n; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1411 continue; |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1412 } |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1413 else if (isspace(c)) |
419
89e40d43065d
Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
418
diff
changeset
|
1414 continue; // Skip spaces. |
496
5a607597dc22
Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
494
diff
changeset
|
1415 else if (isEOF(c)) |
53
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1416 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1417 error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedHexString); |
53
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1418 t.pf = 0; |
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1419 goto Lreturn; |
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1420 } |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1421 else |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1422 { |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1423 auto errorAt = p; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1424 if (!isascii(c)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1425 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1426 c = decodeUTF8(); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1427 if (isUnicodeNewlineChar(c)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1428 goto case '\n'; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1429 } |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1430 error(errorAt, MID.NonHexCharInHexString, cast(dchar)c); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1431 } |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1432 } |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1433 } |
53
1786c2825491
- Fixed scanner for hex string literals. Terminating string with 0. Relocated some code to the bottom.
aziz
parents:
52
diff
changeset
|
1434 assert(0); |
31
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1435 } |
94f09f4e988e
- Added struct for strings to Token with 'pf' = postfix.
aziz
parents:
30
diff
changeset
|
1436 |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1437 version(D2) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1438 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1439 void scanDelimitedStringLiteral(ref Token t) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1440 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1441 assert(p[0] == 'q' && p[1] == '"'); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1442 t.kind = TOK.String; |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1443 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1444 auto tokenLineNum = lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1445 auto tokenLineBegin = lineBegin; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1446 |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1447 char[] buffer; |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1448 dchar opening_delim = 0, // 0 if no nested delimiter or '[', '(', '<', '{' |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1449 closing_delim; // Will be ']', ')', '>', '}, |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1450 // the first character of an identifier or |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1451 // any other Unicode/ASCII character. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1452 char[] str_delim; // Identifier delimiter. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1453 uint level = 1; // Counter for nestable delimiters. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1454 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1455 ++p; ++p; // Skip q" |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1456 uint c = *p; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1457 switch (c) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1458 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1459 case '(': |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1460 opening_delim = c; |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1461 closing_delim = ')'; // c + 1 |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1462 break; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1463 case '[', '<', '{': |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1464 opening_delim = c; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1465 closing_delim = c + 2; // Get to closing counterpart. Feature of ASCII table. |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1466 break; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1467 default: |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1468 dchar scanNewline() |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1469 { |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1470 switch (*p) |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1471 { |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1472 case '\r': |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1473 if (p[1] == '\n') |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1474 ++p; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1475 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1476 assert(isNewlineEnd(p)); |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1477 ++p; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1478 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1479 setLineBegin(p); |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1480 return '\n'; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1481 default: |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1482 if (isUnicodeNewline(p)) |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1483 { |
419
89e40d43065d
Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
418
diff
changeset
|
1484 ++p; ++p; |
89e40d43065d
Added new member 'p_newl' to Lexer and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
418
diff
changeset
|
1485 goto case '\n'; |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1486 } |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1487 } |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1488 return 0; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1489 } |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1490 // Skip leading newlines: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1491 while (scanNewline() != 0) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1492 {} |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1493 assert(!isNewline(p)); |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1494 |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1495 char* begin = p; |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1496 c = *p; |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1497 closing_delim = c; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1498 // TODO: Check for non-printable characters? |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1499 if (!isascii(c)) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1500 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1501 closing_delim = decodeUTF8(); |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1502 if (!isUniAlpha(closing_delim)) |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1503 break; // Not an identifier. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1504 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1505 else if (!isidbeg(c)) |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1506 break; // Not an identifier. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1507 |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1508 // Parse Identifier + EndOfLine |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1509 do |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1510 { c = *++p; } |
550
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
1511 while (isident(c) || !isascii(c) && isUnicodeAlpha()) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1512 // Store identifier |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1513 str_delim = begin[0..p-begin]; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1514 // Scan newline |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1515 if (scanNewline() == '\n') |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1516 --p; // Go back one because of "c = *++p;" in main loop. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1517 else |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1518 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1519 // TODO: error(p, MID.ExpectedNewlineAfterIdentDelim); |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1520 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1521 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1522 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1523 bool checkStringDelim(char* p) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1524 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1525 assert(str_delim.length != 0); |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1526 if (buffer[$-1] == '\n' && // Last character copied to buffer must be '\n'. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1527 end-p >= str_delim.length && // Check remaining length. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1528 p[0..str_delim.length] == str_delim) // Compare. |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1529 return true; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1530 return false; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1531 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1532 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1533 while (1) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1534 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1535 c = *++p; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1536 switch (c) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1537 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1538 case '\r': |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1539 if (p[1] == '\n') |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1540 ++p; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1541 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1542 assert(isNewlineEnd(p)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1543 c = '\n'; // Convert Newline to '\n'. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1544 ++lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1545 setLineBegin(p+1); |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1546 break; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1547 case 0, _Z_: |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1548 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedDelimitedString); |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1549 goto Lreturn3; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1550 default: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1551 if (!isascii(c)) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1552 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1553 auto begin = p; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1554 c = decodeUTF8(); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1555 if (isUnicodeNewlineChar(c)) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1556 goto case '\n'; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1557 if (c == closing_delim) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1558 { |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1559 if (str_delim.length) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1560 { |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1561 if (checkStringDelim(begin)) |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1562 { |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1563 p = begin + str_delim.length; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1564 goto Lreturn2; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1565 } |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1566 } |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1567 else |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1568 { |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1569 assert(level == 1); |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1570 --level; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1571 goto Lreturn; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1572 } |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1573 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1574 encodeUTF8(buffer, c); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1575 continue; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1576 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1577 else |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1578 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1579 if (c == opening_delim) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1580 ++level; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1581 else if (c == closing_delim) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1582 { |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1583 if (str_delim.length) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1584 { |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1585 if (checkStringDelim(p)) |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1586 { |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1587 p += str_delim.length; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1588 goto Lreturn2; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1589 } |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1590 } |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1591 else if (--level == 0) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1592 goto Lreturn; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1593 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1594 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1595 } |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
1596 assert(isascii(c)); |
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
1597 buffer ~= c; |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1598 } |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1599 Lreturn: // Character delimiter. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1600 assert(c == closing_delim); |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1601 assert(level == 0); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1602 ++p; // Skip closing delimiter. |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1603 Lreturn2: // String delimiter. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1604 if (*p == '"') |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1605 ++p; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1606 else |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1607 { |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1608 // TODO: error(p, MID.ExpectedDblQuoteAfterDelim, str_delim.length ? str_delim : closing_delim~""); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1609 } |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1610 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1611 t.pf = scanPostfix(); |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1612 Lreturn3: // Error. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1613 t.str = buffer ~ '\0'; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1614 t.end = p; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1615 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1616 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1617 void scanTokenStringLiteral(ref Token t) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1618 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1619 assert(p[0] == 'q' && p[1] == '{'); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1620 t.kind = TOK.String; |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1621 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1622 auto tokenLineNum = lineNum; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1623 auto tokenLineBegin = lineBegin; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1624 |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1625 // A guard against changes to particular members: |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1626 // this.lineNum_hline and this.errorPath |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1627 ++inTokenString; |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1628 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1629 uint lineNum = this.lineNum; |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1630 uint level = 1; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1631 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1632 ++p; ++p; // Skip q{ |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1633 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1634 auto prev_t = &t; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1635 Token* token; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1636 while (1) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1637 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1638 token = new Token; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1639 scan(*token); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1640 // Save the tokens in a doubly linked list. |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1641 // Could be useful for various tools. |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1642 token.prev = prev_t; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1643 prev_t.next = token; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1644 prev_t = token; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1645 switch (token.kind) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1646 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1647 case TOK.LBrace: |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1648 ++level; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1649 continue; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1650 case TOK.RBrace: |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1651 if (--level == 0) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1652 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1653 t.tok_str = t.next; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1654 t.next = null; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1655 break; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1656 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1657 continue; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1658 case TOK.EOF: |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1659 // TODO: error(tokenLineNum, tokenLineBegin, t.start, MID.UnterminatedTokenString); |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1660 t.tok_str = t.next; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1661 t.next = token; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1662 break; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1663 default: |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1664 continue; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1665 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1666 break; // Exit loop. |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1667 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1668 |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1669 assert(token.kind == TOK.RBrace || token.kind == TOK.EOF); |
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1670 assert(token.kind == TOK.RBrace && t.next is null || |
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1671 token.kind == TOK.EOF && t.next !is null); |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1672 |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1673 char[] buffer; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1674 // token points to } or EOF |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
1675 if (token.kind == TOK.EOF) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1676 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1677 t.end = token.start; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1678 buffer = t.srcText[2..$].dup ~ '\0'; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1679 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1680 else |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1681 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1682 // Assign to buffer before scanPostfix(). |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1683 t.end = p; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1684 buffer = t.srcText[2..$-1].dup ~ '\0'; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1685 t.pf = scanPostfix(); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1686 t.end = p; // Assign again because of postfix. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1687 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1688 // Convert newlines to '\n'. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1689 if (lineNum != this.lineNum) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1690 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1691 assert(buffer[$-1] == '\0'); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1692 uint i, j; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1693 for (; i < buffer.length; ++i) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1694 switch (buffer[i]) |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1695 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1696 case '\r': |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1697 if (buffer[i+1] == '\n') |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1698 ++i; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1699 case '\n': |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1700 assert(isNewlineEnd(buffer.ptr + i)); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1701 buffer[j++] = '\n'; // Convert Newline to '\n'. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1702 break; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1703 default: |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1704 if (isUnicodeNewline(buffer.ptr + i)) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1705 { |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1706 ++i; ++i; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1707 goto case '\n'; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1708 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1709 buffer[j++] = buffer[i]; // Copy. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1710 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1711 buffer.length = j; // Adjust length. |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1712 } |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1713 assert(buffer[$-1] == '\0'); |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1714 t.str = buffer; |
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1715 |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1716 --inTokenString; |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1717 } |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1718 } // version(D2) |
383
6a5fc22cae34
Implemented scanner for new string literals and applied some fixes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
382
diff
changeset
|
1719 |
45 | 1720 dchar scanEscapeSequence() |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1721 out(result) |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
1722 { assert(isValidChar(result)); } |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1723 body |
45 | 1724 { |
386
392a0068fc61
Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
384
diff
changeset
|
1725 assert(*p == '\\'); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1726 |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1727 auto sequenceStart = p; // Used for error reporting. |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1728 |
386
392a0068fc61
Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
384
diff
changeset
|
1729 ++p; |
45 | 1730 uint c = char2ev(*p); |
82
fc645fb2fe72
- scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents:
71
diff
changeset
|
1731 if (c) |
fc645fb2fe72
- scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents:
71
diff
changeset
|
1732 { |
46 | 1733 ++p; |
45 | 1734 return c; |
46 | 1735 } |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1736 |
45 | 1737 uint digits = 2; |
1738 | |
1739 switch (*p) | |
1740 { | |
1741 case 'x': | |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1742 assert(c == 0); |
45 | 1743 while (1) |
1744 { | |
1745 ++p; | |
1746 if (ishexad(*p)) | |
1747 { | |
1748 c *= 16; | |
1749 if (*p <= '9') | |
48 | 1750 c += *p - '0'; |
45 | 1751 else if (*p <= 'F') |
48 | 1752 c += *p - 'A' + 10; |
45 | 1753 else |
48 | 1754 c += *p - 'a' + 10; |
82
fc645fb2fe72
- scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents:
71
diff
changeset
|
1755 |
fc645fb2fe72
- scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents:
71
diff
changeset
|
1756 if (!--digits) |
fc645fb2fe72
- scanEscapeSequences() doesn't return 0xFFFF as an error value anymore, because it is a valid codepoint usable by the user.
aziz
parents:
71
diff
changeset
|
1757 { |
48 | 1758 ++p; |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
1759 if (isValidChar(c)) |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1760 return c; // Return valid escape value. |
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1761 |
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1762 error(sequenceStart, MID.InvalidUnicodeEscapeSequence, sequenceStart[0..p-sequenceStart]); |
45 | 1763 break; |
48 | 1764 } |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1765 continue; |
45 | 1766 } |
432
3ead178e0662
Added message MID.InvalidUnicodeEscapeSequence.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
430
diff
changeset
|
1767 |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1768 error(sequenceStart, MID.InsufficientHexDigits); |
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1769 break; |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
1770 } |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1771 break; |
45 | 1772 case 'u': |
1773 digits = 4; | |
1774 goto case 'x'; | |
1775 case 'U': | |
1776 digits = 8; | |
1777 goto case 'x'; | |
1778 default: | |
48 | 1779 if (isoctal(*p)) |
45 | 1780 { |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1781 assert(c == 0); |
48 | 1782 c += *p - '0'; |
1783 ++p; | |
1784 if (!isoctal(*p)) | |
1785 return c; | |
1786 c *= 8; | |
1787 c += *p - '0'; | |
1788 ++p; | |
1789 if (!isoctal(*p)) | |
1790 return c; | |
1791 c *= 8; | |
1792 c += *p - '0'; | |
1793 ++p; | |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1794 return c; // Return valid escape value. |
48 | 1795 } |
1796 else if(*p == '&') | |
1797 { | |
1798 if (isalpha(*++p)) | |
45 | 1799 { |
272
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1800 auto begin = p; |
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1801 while (isalnum(*++p)) |
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1802 {} |
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1803 |
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1804 if (*p == ';') |
48 | 1805 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1806 // Pass entity excluding '&' and ';'. |
272
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1807 c = entity2Unicode(begin[0..p - begin]); |
386
392a0068fc61
Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
384
diff
changeset
|
1808 ++p; // Skip ; |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1809 if (c != 0xFFFF) |
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1810 return c; // Return valid escape value. |
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1811 else |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1812 error(sequenceStart, MID.UndefinedHTMLEntity, sequenceStart[0 .. p - sequenceStart]); |
272
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1813 } |
0bde32503976
- Added module HtmlEntities. It contains a table for converting HTML entities to Unicode characters.
aziz
parents:
249
diff
changeset
|
1814 else |
432
3ead178e0662
Added message MID.InvalidUnicodeEscapeSequence.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
430
diff
changeset
|
1815 error(sequenceStart, MID.UnterminatedHTMLEntity, sequenceStart[0 .. p - sequenceStart]); |
45 | 1816 } |
48 | 1817 else |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1818 error(sequenceStart, MID.InvalidBeginHTMLEntity); |
45 | 1819 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1820 else if (isEndOfLine(p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1821 error(sequenceStart, MID.UndefinedEscapeSequence, |
496
5a607597dc22
Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
494
diff
changeset
|
1822 isEOF(*p) ? `\EOF` : `\NewLine`); |
45 | 1823 else |
386
392a0068fc61
Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
384
diff
changeset
|
1824 { |
392a0068fc61
Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
384
diff
changeset
|
1825 char[] str = `\`; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1826 if (isascii(c)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1827 str ~= *p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
1828 else |
387
ad0cbd1c8881
Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
386
diff
changeset
|
1829 encodeUTF8(str, decodeUTF8()); |
ad0cbd1c8881
Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
386
diff
changeset
|
1830 ++p; |
ad0cbd1c8881
Undefined escape sequences are passed to error() now.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
386
diff
changeset
|
1831 // TODO: check for unprintable character? |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1832 error(sequenceStart, MID.UndefinedEscapeSequence, str); |
386
392a0068fc61
Refactored code related to scanning escape sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
384
diff
changeset
|
1833 } |
45 | 1834 } |
433
063cd016c913
Fixed some issues in scanEscapeSequence().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
432
diff
changeset
|
1835 return REPLACEMENT_CHAR; // Error: return replacement character. |
45 | 1836 } |
1837 | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1838 /* |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1839 IntegerLiteral:= (Dec|Hex|Bin|Oct)Suffix? |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1840 Dec:= (0|[1-9][0-9_]*) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1841 Hex:= 0[xX] HexDigits |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1842 Bin:= 0[bB][01_]+ |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1843 Oct:= 0[0-7_]+ |
68
7eb83dd38901
- Simplified suffix rule and added a few more numbers to unittest.
aziz
parents:
67
diff
changeset
|
1844 Suffix:= (L[uU]?|[uU]L?) |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1845 HexDigits:= [0-9a-zA-Z_]+ |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1846 |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1847 Invalid: "0b_", "0x_", "._" |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1848 */ |
15 | 1849 void scanNumber(ref Token t) |
1850 { | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1851 ulong ulong_; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1852 bool overflow; |
58 | 1853 bool isDecimal; |
57
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1854 size_t digits; |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1855 |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1856 if (*p != '0') |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1857 goto LscanInteger; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1858 ++p; // skip zero |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1859 // check for xX bB ... |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1860 switch (*p) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1861 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1862 case 'x','X': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1863 goto LscanHex; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1864 case 'b','B': |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1865 goto LscanBinary; |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1866 case 'L': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1867 if (p[1] == 'i') |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1868 goto LscanReal; // 0Li |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1869 break; // 0L |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1870 case '.': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1871 if (p[1] == '.') |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1872 break; // 0.. |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1873 // 0. |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1874 case 'i','f','F', // Imaginary and float literal suffixes. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
1875 'e', 'E': // Float exponent. |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1876 goto LscanReal; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1877 default: |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1878 if (*p == '_') |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1879 goto LscanOctal; // 0_ |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1880 else if (isdigit(*p)) |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1881 { |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1882 if (*p == '8' || *p == '9') |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1883 goto Loctal_hasDecimalDigits; // 08 or 09 |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1884 else |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1885 goto Loctal_enter_loop; // 0[0-7] |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1886 } |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1887 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1888 |
323 | 1889 // Number 0 |
1890 assert(p[-1] == '0'); | |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1891 assert(*p != '_' && !isdigit(*p)); |
323 | 1892 assert(ulong_ == 0); |
58 | 1893 isDecimal = true; |
1894 goto Lfinalize; | |
1895 | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1896 LscanInteger: |
323 | 1897 assert(*p != 0 && isdigit(*p)); |
58 | 1898 isDecimal = true; |
323 | 1899 goto Lenter_loop_int; |
63 | 1900 while (1) |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1901 { |
63 | 1902 if (*++p == '_') |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1903 continue; |
63 | 1904 if (!isdigit(*p)) |
1905 break; | |
323 | 1906 Lenter_loop_int: |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1907 if (ulong_ < ulong.max/10 || (ulong_ == ulong.max/10 && *p <= '5')) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1908 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1909 ulong_ *= 10; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1910 ulong_ += *p - '0'; |
58 | 1911 continue; |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1912 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1913 // Overflow: skip following digits. |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1914 overflow = true; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1915 while (isdigit(*++p)) {} |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1916 break; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1917 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1918 |
61 | 1919 // The number could be a float, so check overflow below. |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1920 switch (*p) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1921 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1922 case '.': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1923 if (p[1] != '.') |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1924 goto LscanReal; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1925 break; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1926 case 'L': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1927 if (p[1] != 'i') |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1928 break; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1929 case 'i', 'f', 'F', 'e', 'E': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1930 goto LscanReal; |
58 | 1931 default: |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1932 } |
58 | 1933 |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1934 if (overflow) |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1935 error(t.start, MID.OverflowDecimalNumber); |
58 | 1936 |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1937 assert((isdigit(p[-1]) || p[-1] == '_') && !isdigit(*p) && *p != '_'); |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1938 goto Lfinalize; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1939 |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1940 LscanHex: |
61 | 1941 assert(digits == 0); |
410
4d9ee8e60712
Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
403
diff
changeset
|
1942 assert(*p == 'x' || *p == 'X'); |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1943 while (1) |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1944 { |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1945 if (*++p == '_') |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1946 continue; |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1947 if (!ishexad(*p)) |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1948 break; |
61 | 1949 ++digits; |
57
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1950 ulong_ *= 16; |
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1951 if (*p <= '9') |
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1952 ulong_ += *p - '0'; |
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1953 else if (*p <= 'F') |
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1954 ulong_ += *p - 'A' + 10; |
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1955 else |
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1956 ulong_ += *p - 'a' + 10; |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1957 } |
57
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1958 |
410
4d9ee8e60712
Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
403
diff
changeset
|
1959 assert(ishexad(p[-1]) || p[-1] == '_' || p[-1] == 'x' || p[-1] == 'X'); |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
1960 assert(!ishexad(*p) && *p != '_'); |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
1961 |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1962 switch (*p) |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1963 { |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1964 case '.': |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
1965 if (p[1] == '.') |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1966 break; |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
1967 case 'p', 'P': |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
1968 return scanHexReal(t); |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1969 default: |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
1970 } |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
1971 |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1972 if (digits == 0 || digits > 16) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1973 error(t.start, digits == 0 ? MID.NoDigitsInHexNumber : MID.OverflowHexNumber); |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
1974 |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1975 goto Lfinalize; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1976 |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
1977 LscanBinary: |
57
c0f1c8be3a47
- Added code for converting hex characters to binary numbers.
aziz
parents:
56
diff
changeset
|
1978 assert(digits == 0); |
410
4d9ee8e60712
Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
403
diff
changeset
|
1979 assert(*p == 'b' || *p == 'B'); |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1980 while (1) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1981 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1982 if (*++p == '0') |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1983 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1984 ++digits; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1985 ulong_ *= 2; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1986 } |
367
dda55fae37de
- ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents:
365
diff
changeset
|
1987 else if (*p == '1') |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1988 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1989 ++digits; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1990 ulong_ *= 2; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1991 ulong_ += *p - '0'; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1992 } |
367
dda55fae37de
- ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents:
365
diff
changeset
|
1993 else if (*p == '_') |
dda55fae37de
- ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents:
365
diff
changeset
|
1994 continue; |
dda55fae37de
- ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents:
365
diff
changeset
|
1995 else |
dda55fae37de
- ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents:
365
diff
changeset
|
1996 break; |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1997 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
1998 |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
1999 if (digits == 0 || digits > 64) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2000 error(t.start, digits == 0 ? MID.NoDigitsInBinNumber : MID.OverflowBinaryNumber); |
59
3e594725899a
- Issuing error when no digits were found in hex and binary numbers.
aziz
parents:
58
diff
changeset
|
2001 |
410
4d9ee8e60712
Added destructors for two particular tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
403
diff
changeset
|
2002 assert(p[-1] == '0' || p[-1] == '1' || p[-1] == '_' || p[-1] == 'b' || p[-1] == 'B', p[-1] ~ ""); |
367
dda55fae37de
- ImportGraph.execute() can parse all modules depending on the imports of the root module.
aziz
parents:
365
diff
changeset
|
2003 assert( !(*p == '0' || *p == '1' || *p == '_') ); |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2004 goto Lfinalize; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2005 |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
2006 LscanOctal: |
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
2007 assert(*p == '_'); |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2008 while (1) |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2009 { |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2010 if (*++p == '_') |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2011 continue; |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2012 if (!isoctal(*p)) |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2013 break; |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
2014 Loctal_enter_loop: |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2015 if (ulong_ < ulong.max/2 || (ulong_ == ulong.max/2 && *p <= '1')) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2016 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2017 ulong_ *= 8; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2018 ulong_ += *p - '0'; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2019 continue; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2020 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2021 // Overflow: skip following digits. |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2022 overflow = true; |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
2023 while (isoctal(*++p)) {} |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2024 break; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2025 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2026 |
63 | 2027 bool hasDecimalDigits; |
2028 if (isdigit(*p)) | |
2029 { | |
396
0a4619735ce9
Applied fixes to Lexer, Parser and other classes.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
393
diff
changeset
|
2030 Loctal_hasDecimalDigits: |
63 | 2031 hasDecimalDigits = true; |
2032 while (isdigit(*++p)) {} | |
2033 } | |
2034 | |
2035 // The number could be a float, so check errors below. | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2036 switch (*p) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2037 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2038 case '.': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2039 if (p[1] != '.') |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2040 goto LscanReal; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2041 break; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2042 case 'L': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2043 if (p[1] != 'i') |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2044 break; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2045 case 'i', 'f', 'F', 'e', 'E': |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2046 goto LscanReal; |
63 | 2047 default: |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2048 } |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2049 |
63 | 2050 if (hasDecimalDigits) |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2051 error(t.start, MID.OctalNumberHasDecimals); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2052 |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2053 if (overflow) |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2054 error(t.start, MID.OverflowOctalNumber); |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2055 // goto Lfinalize; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2056 |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2057 Lfinalize: |
58 | 2058 enum Suffix |
2059 { | |
2060 None = 0, | |
2061 Unsigned = 1, | |
60 | 2062 Long = 2 |
58 | 2063 } |
60 | 2064 |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2065 // Scan optional suffix: L, Lu, LU, u, uL, U or UL. |
58 | 2066 Suffix suffix; |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2067 while (1) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2068 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2069 switch (*p) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2070 { |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2071 case 'L': |
58 | 2072 if (suffix & Suffix.Long) |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2073 break; |
60 | 2074 suffix |= Suffix.Long; |
2075 ++p; | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2076 continue; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2077 case 'u', 'U': |
58 | 2078 if (suffix & Suffix.Unsigned) |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2079 break; |
60 | 2080 suffix |= Suffix.Unsigned; |
2081 ++p; | |
2082 continue; | |
58 | 2083 default: |
2084 break; | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2085 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2086 break; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2087 } |
58 | 2088 |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2089 // Determine type of Integer. |
58 | 2090 switch (suffix) |
2091 { | |
2092 case Suffix.None: | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2093 if (ulong_ & 0x8000_0000_0000_0000) |
58 | 2094 { |
2095 if (isDecimal) | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2096 error(t.start, MID.OverflowDecimalSign); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2097 t.kind = TOK.Uint64; |
58 | 2098 } |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2099 else if (ulong_ & 0xFFFF_FFFF_0000_0000) |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2100 t.kind = TOK.Int64; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2101 else if (ulong_ & 0x8000_0000) |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2102 t.kind = isDecimal ? TOK.Int64 : TOK.Uint32; |
58 | 2103 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2104 t.kind = TOK.Int32; |
58 | 2105 break; |
2106 case Suffix.Unsigned: | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2107 if (ulong_ & 0xFFFF_FFFF_0000_0000) |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2108 t.kind = TOK.Uint64; |
58 | 2109 else |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2110 t.kind = TOK.Uint32; |
58 | 2111 break; |
2112 case Suffix.Long: | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2113 if (ulong_ & 0x8000_0000_0000_0000) |
58 | 2114 { |
2115 if (isDecimal) | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2116 error(t.start, MID.OverflowDecimalSign); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2117 t.kind = TOK.Uint64; |
58 | 2118 } |
2119 else | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2120 t.kind = TOK.Int64; |
58 | 2121 break; |
2122 case Suffix.Unsigned | Suffix.Long: | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2123 t.kind = TOK.Uint64; |
58 | 2124 break; |
2125 default: | |
2126 assert(0); | |
2127 } | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2128 t.ulong_ = ulong_; |
15 | 2129 t.end = p; |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2130 return; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2131 LscanReal: |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2132 scanReal(t); |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2133 return; |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2134 } |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2135 |
63 | 2136 /* |
2137 FloatLiteral:= Float[fFL]?i? | |
2138 Float:= DecFloat | HexFloat | |
67
996065105910
- Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents:
66
diff
changeset
|
2139 DecFloat:= ([0-9][0-9_]*[.][0-9_]*DecExponent?) | [.][0-9][0-9_]*DecExponent? | [0-9][0-9_]*DecExponent |
63 | 2140 DecExponent:= [eE][+-]?[0-9][0-9_]* |
2141 HexFloat:= 0[xX](HexDigits[.]HexDigits | [.][0-9a-zA-Z]HexDigits? | HexDigits)HexExponent | |
2142 HexExponent:= [pP][+-]?[0-9][0-9_]* | |
2143 */ | |
56
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2144 void scanReal(ref Token t) |
63af7ddf52e1
- Started properly implementing number scanner. Added stub for real numbers.
aziz
parents:
55
diff
changeset
|
2145 { |
63 | 2146 if (*p == '.') |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2147 { |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2148 assert(p[1] != '.'); |
63 | 2149 // This function was called by scan() or scanNumber(). |
2150 while (isdigit(*++p) || *p == '_') {} | |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2151 } |
63 | 2152 else |
2153 // This function was called by scanNumber(). | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2154 assert(delegate () |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2155 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2156 switch (*p) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2157 { |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2158 case 'L': |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2159 if (p[1] != 'i') |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2160 return false; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2161 case 'i', 'f', 'F', 'e', 'E': |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2162 return true; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2163 default: |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2164 } |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2165 return false; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2166 }() |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2167 ); |
63 | 2168 |
2169 // Scan exponent. | |
2170 if (*p == 'e' || *p == 'E') | |
2171 { | |
2172 ++p; | |
2173 if (*p == '-' || *p == '+') | |
2174 ++p; | |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2175 if (isdigit(*p)) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2176 while (isdigit(*++p) || *p == '_') {} |
63 | 2177 else |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2178 error(t.start, MID.FloatExpMustStartWithDigit); |
63 | 2179 } |
2180 | |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2181 // Copy whole number and remove underscores from buffer. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2182 char[] buffer = t.start[0..p-t.start].dup; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2183 uint j; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2184 foreach (c; buffer) |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2185 if (c != '_') |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2186 buffer[j++] = c; |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2187 buffer.length = j; // Adjust length. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2188 buffer ~= 0; // Terminate for C functions. |
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2189 |
63 | 2190 finalizeFloat(t, buffer); |
30 | 2191 } |
2192 | |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2193 void scanHexReal(ref Token t) |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2194 { |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2195 assert(*p == '.' || *p == 'p' || *p == 'P'); |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2196 MID mid; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2197 if (*p == '.') |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2198 while (ishexad(*++p) || *p == '_') |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2199 {} |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2200 // Decimal exponent is required. |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2201 if (*p != 'p' && *p != 'P') |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2202 { |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2203 mid = MID.HexFloatExponentRequired; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2204 goto Lerr; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2205 } |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2206 // Scan exponent |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2207 assert(*p == 'p' || *p == 'P'); |
393
fce1e6133dac
Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
392
diff
changeset
|
2208 ++p; |
fce1e6133dac
Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
392
diff
changeset
|
2209 if (*p == '+' || *p == '-') |
fce1e6133dac
Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
392
diff
changeset
|
2210 ++p; |
fce1e6133dac
Applied fix to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
392
diff
changeset
|
2211 if (!isdigit(*p)) |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2212 { |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2213 mid = MID.HexFloatExpMustStartWithDigit; |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2214 goto Lerr; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2215 } |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2216 while (isdigit(*++p) || *p == '_') |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2217 {} |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2218 // Copy whole number and remove underscores from buffer. |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2219 char[] buffer = t.start[0..p-t.start].dup; |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2220 uint j; |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2221 foreach (c; buffer) |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2222 if (c != '_') |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2223 buffer[j++] = c; |
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2224 buffer.length = j; // Adjust length. |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2225 buffer ~= 0; // Terminate for C functions. |
63 | 2226 finalizeFloat(t, buffer); |
2227 return; | |
2228 Lerr: | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2229 t.kind = TOK.Float32; |
63 | 2230 t.end = p; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2231 error(t.start, mid); |
63 | 2232 } |
2233 | |
2234 void finalizeFloat(ref Token t, string buffer) | |
2235 { | |
389
c4bfceab7246
Applied fixes and improvements to hex float scanner.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
388
diff
changeset
|
2236 assert(buffer[$-1] == 0); |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2237 // Float number is well-formed. Check suffixes and do conversion. |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2238 switch (*p) |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2239 { |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2240 case 'f', 'F': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2241 t.kind = TOK.Float32; |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2242 t.float_ = strtof(buffer.ptr, null); |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2243 ++p; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2244 break; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2245 case 'L': |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2246 t.kind = TOK.Float80; |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2247 t.real_ = strtold(buffer.ptr, null); |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2248 ++p; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2249 break; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2250 default: |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2251 t.kind = TOK.Float64; |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2252 t.double_ = strtod(buffer.ptr, null); |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2253 } |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2254 if (*p == 'i') |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2255 { |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2256 ++p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2257 t.kind += 3; // Switch to imaginary counterpart. |
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2258 assert(t.kind == TOK.Imaginary32 || |
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2259 t.kind == TOK.Imaginary64 || |
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2260 t.kind == TOK.Imaginary80); |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2261 } |
391
33b566df6af4
Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
390
diff
changeset
|
2262 if (errno() == ERANGE) |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2263 error(t.start, MID.OverflowFloatNumber); |
62
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2264 t.end = p; |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2265 } |
96af5653acef
- Fixed loop of hex number scanner. Moved checks under the switch block.
aziz
parents:
61
diff
changeset
|
2266 |
30 | 2267 /// Scan special token: #line Integer [Filespec] EndOfLine |
360
b6a3755eba94
- Renamed scanSpecialToken() to scanSpecialTokenSequence().
aziz
parents:
350
diff
changeset
|
2268 void scanSpecialTokenSequence(ref Token t) |
30 | 2269 { |
47
8aa37a78937b
- Properly implemented scanner for normal string literals.
aziz
parents:
46
diff
changeset
|
2270 assert(*p == '#'); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2271 t.kind = TOK.HashLine; |
552
3bc7801c207e
Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
550
diff
changeset
|
2272 t.setWhitespaceFlag(); |
323 | 2273 |
30 | 2274 MID mid; |
762
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2275 char* errorAtColumn = p; |
30 | 2276 |
323 | 2277 ++p; |
762
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2278 if (!(p[0] == 'l' && p[1] == 'i' && p[2] == 'n' && p[3] == 'e')) |
51 | 2279 { |
370 | 2280 mid = MID.ExpectedIdentifierSTLine; |
51 | 2281 goto Lerr; |
2282 } | |
323 | 2283 p += 3; |
2284 | |
388
ae154eceba65
Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
387
diff
changeset
|
2285 // TODO: #line58"path/file" is legal. Require spaces? |
ae154eceba65
Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
387
diff
changeset
|
2286 // State.Space could be used for that purpose. |
323 | 2287 enum State |
388
ae154eceba65
Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
387
diff
changeset
|
2288 { /+Space,+/ Integer, Filespec, End } |
51 | 2289 |
388
ae154eceba65
Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
387
diff
changeset
|
2290 State state = State.Integer; |
762
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2291 char* tokenEnd = p + 1; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2292 while (!isEndOfLine(++p)) |
51 | 2293 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2294 if (isspace(*p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2295 continue; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2296 if (state == State.Integer) |
323 | 2297 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2298 if (!isdigit(*p)) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2299 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2300 errorAtColumn = p; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2301 mid = MID.ExpectedIntegerAfterSTLine; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2302 goto Lerr; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2303 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2304 t.tokLineNum = new Token; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2305 scan(*t.tokLineNum); |
762
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2306 tokenEnd = p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2307 if (t.tokLineNum.kind != TOK.Int32 && t.tokLineNum.kind != TOK.Uint32) |
323 | 2308 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2309 errorAtColumn = t.tokLineNum.start; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2310 mid = MID.ExpectedIntegerAfterSTLine; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2311 goto Lerr; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2312 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2313 --p; // Go one back because scan() advanced p past the integer. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2314 state = State.Filespec; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2315 } |
762
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2316 else if (state == State.Filespec && *p == '"') |
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2317 { // MID.ExpectedFilespec is eprecated. |
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2318 // if (*p != '"') |
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2319 // { |
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2320 // errorAtColumn = p; |
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2321 // mid = MID.ExpectedFilespec; |
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2322 // goto Lerr; |
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2323 // } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2324 t.tokLineFilespec = new Token; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2325 t.tokLineFilespec.start = p; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2326 t.tokLineFilespec.kind = TOK.Filespec; |
552
3bc7801c207e
Refactored the way how tokens are flagged as whitespace.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
550
diff
changeset
|
2327 t.tokLineFilespec.setWhitespaceFlag(); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2328 while (*++p != '"') |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2329 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2330 if (isEndOfLine(p)) |
323 | 2331 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2332 errorAtColumn = t.tokLineFilespec.start; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2333 mid = MID.UnterminatedFilespec; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2334 t.tokLineFilespec.end = p; |
762
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2335 tokenEnd = p; |
323 | 2336 goto Lerr; |
2337 } | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2338 isascii(*p) || decodeUTF8(); |
323 | 2339 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2340 auto start = t.tokLineFilespec.start +1; // +1 skips '"' |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2341 t.tokLineFilespec.str = start[0 .. p - start]; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2342 t.tokLineFilespec.end = p + 1; |
762
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2343 tokenEnd = p + 1; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2344 state = State.End; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2345 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2346 else/+ if (state == State.End)+/ |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2347 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2348 mid = MID.UnterminatedSpecialToken; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2349 goto Lerr; |
323 | 2350 } |
51 | 2351 } |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2352 assert(isEndOfLine(p)); |
323 | 2353 |
388
ae154eceba65
Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
387
diff
changeset
|
2354 if (state == State.Integer) |
51 | 2355 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2356 errorAtColumn = p; |
388
ae154eceba65
Applied some fixes to scanning and printing #line tokens.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
387
diff
changeset
|
2357 mid = MID.ExpectedIntegerAfterSTLine; |
30 | 2358 goto Lerr; |
2359 } | |
2360 | |
392
bb935c6f9b7a
Applied fixes and improvements to the Lexer class.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
391
diff
changeset
|
2361 // Evaluate #line only when not in token string. |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2362 if (!inTokenString && t.tokLineNum) |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2363 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2364 this.lineNum_hline = this.lineNum - t.tokLineNum.uint_ + 1; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2365 if (t.tokLineFilespec) |
607
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
2366 newFilePath(t.tokLineFilespec.str); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2367 } |
762
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2368 p = tokenEnd; |
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2369 t.end = tokenEnd; |
30 | 2370 |
2371 return; | |
2372 Lerr: | |
762
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2373 p = tokenEnd; |
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2374 t.end = tokenEnd; |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2375 error(errorAtColumn, mid); |
371 | 2376 } |
2377 | |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2378 /++ |
414
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2379 Insert an empty dummy token before t. |
737
f88b5285b86b
Implemented DDocEmitter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
715
diff
changeset
|
2380 |
414
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2381 Useful in the parsing phase for representing a node in the AST |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2382 that doesn't consume an actual token from the source text. |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2383 +/ |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2384 Token* insertEmptyTokenBefore(Token* t) |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2385 { |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2386 assert(t !is null && t.prev !is null); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2387 assert(text.ptr <= t.start && t.start < end, Token.toString(t.kind)); |
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2388 assert(text.ptr <= t.end && t.end <= end, Token.toString(t.kind)); |
414
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2389 |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2390 auto prev_t = t.prev; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2391 auto new_t = new Token; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2392 new_t.kind = TOK.Empty; |
414
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2393 new_t.start = new_t.end = prev_t.end; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2394 // Link in new token. |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2395 prev_t.next = new_t; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2396 new_t.prev = prev_t; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2397 new_t.next = t; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2398 t.prev = new_t; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2399 return new_t; |
9c69615a4876
Added method insertEmptyTokenBefore() to Lexer. Made some fixes, too.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
413
diff
changeset
|
2400 } |
105 | 2401 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2402 uint errorLineNumber(uint lineNum) |
239
7911f6a92e6e
- Added 'new' and 'delete' declarations to Token and uncommented next and prev members. Added HEAD to TOK.
aziz
parents:
223
diff
changeset
|
2403 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2404 return lineNum - this.lineNum_hline; |
18
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
2405 } |
c48d2125f1e2
- Moved code for scanning character literals to separate function.
aziz
parents:
17
diff
changeset
|
2406 |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2407 void error(char* columnPos, MID mid, ...) |
207 | 2408 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2409 error_(this.lineNum, this.lineBegin, columnPos, mid, _arguments, _argptr); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2410 } |
207 | 2411 |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2412 void error(uint lineNum, char* lineBegin, char* columnPos, MID mid, ...) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2413 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2414 error_(lineNum, lineBegin, columnPos, mid, _arguments, _argptr); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2415 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2416 |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2417 void error_(uint lineNum, char* lineBegin, char* columnPos, MID mid, |
762
c909a3d3fa52
Fixed vararg issue with gdc.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
755
diff
changeset
|
2418 TypeInfo[] _arguments, Arg _argptr) |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2419 { |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2420 lineNum = this.errorLineNumber(lineNum); |
607
2ed1e6d638cd
Making use of struct NewlineData.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
601
diff
changeset
|
2421 auto errorPath = this.filePaths.setPath; |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2422 auto location = new Location(errorPath, lineNum, lineBegin, columnPos); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2423 auto msg = Format(_arguments, _argptr, GetMsg(mid)); |
520
f203c5248d0b
Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
519
diff
changeset
|
2424 auto error = new LexerError(location, msg); |
f203c5248d0b
Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
519
diff
changeset
|
2425 errors ~= error; |
f203c5248d0b
Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
519
diff
changeset
|
2426 if (infoMan !is null) |
f203c5248d0b
Added 'compile' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
519
diff
changeset
|
2427 infoMan ~= error; |
207 | 2428 } |
2429 | |
306 | 2430 Token* getTokens() |
4 | 2431 { |
306 | 2432 while (nextToken() != TOK.EOF) |
2433 {} | |
2434 return head; | |
3 | 2435 } |
48 | 2436 |
487
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2437 /// Scan the whole text until EOF is encountered. |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2438 void scanAll() |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2439 { |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2440 while (nextToken() != TOK.EOF) |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2441 {} |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2442 } |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2443 |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2444 /// HEAD -> Newline -> First Token |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2445 Token* firstToken() |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2446 { |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2447 return this.head.next.next; |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2448 } |
bccca748d745
Added 'tokenize' command.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
485
diff
changeset
|
2449 |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2450 static void loadKeywords(ref Identifier[string] table) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2451 { |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2452 foreach(k; keywords) |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2453 table[k.str] = k; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2454 } |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2455 |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2456 /// Returns true if str is a valid D identifier. |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2457 static bool isIdentifierString(char[] str) |
365 | 2458 { |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2459 if (str.length == 0 || isdigit(str[0])) |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2460 return false; |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2461 size_t idx; |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2462 do |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2463 { |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2464 auto c = dil.Unicode.decode(str, idx); |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2465 if (c == ERROR_CHAR || !(isident(c) || !isascii(c) && isUniAlpha(c))) |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2466 return false; |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2467 } while (idx < str.length) |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2468 return true; |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2469 } |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2470 |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2471 /// Returns true if str is a keyword or a special token (__FILE__, __LINE__ etc.) |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2472 static bool isReservedIdentifier(char[] str) |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2473 { |
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2474 if (str.length == 0) |
382
66477017cb95
Applied fixes to isNonReservedIdentifier().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
371
diff
changeset
|
2475 return false; |
66477017cb95
Applied fixes to isNonReservedIdentifier().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
371
diff
changeset
|
2476 |
365 | 2477 static Identifier[string] reserved_ids_table; |
2478 if (reserved_ids_table is null) | |
499
52447db67938
Implemented global table of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
498
diff
changeset
|
2479 Lexer.loadKeywords(reserved_ids_table); |
365 | 2480 |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2481 if (!isIdentifierString(str)) |
365 | 2482 return false; |
2483 | |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2484 return (str in reserved_ids_table) !is null; |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
2485 } |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2486 |
550
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2487 /++ |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2488 Returns true if the current character to be decoded is |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2489 a Unicode alpha character. |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2490 The current pointer 'p' is not advanced if false is returned. |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2491 +/ |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2492 bool isUnicodeAlpha() |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2493 { |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2494 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8()."); |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2495 char* p = this.p; |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2496 dchar d = *p; |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2497 ++p; // Move to second byte. |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2498 // Error if second byte is not a trail byte. |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2499 if (!isTrailByte(*p)) |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2500 return false; |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2501 // Check for overlong sequences. |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2502 switch (d) |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2503 { |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2504 case 0xE0, 0xF0, 0xF8, 0xFC: |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2505 if ((*p & d) == 0x80) |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2506 return false; |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2507 default: |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2508 if ((d & 0xFE) == 0xC0) // 1100000x |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2509 return false; |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2510 } |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2511 const char[] checkNextByte = "if (!isTrailByte(*++p))" |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2512 " return false;"; |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2513 const char[] appendSixBits = "d = (d << 6) | *p & 0b0011_1111;"; |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2514 // Decode |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2515 if ((d & 0b1110_0000) == 0b1100_0000) |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2516 { |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2517 d &= 0b0001_1111; |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2518 mixin(appendSixBits); |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2519 } |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2520 else if ((d & 0b1111_0000) == 0b1110_0000) |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2521 { |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2522 d &= 0b0000_1111; |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2523 mixin(appendSixBits ~ |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2524 checkNextByte ~ appendSixBits); |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2525 } |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2526 else if ((d & 0b1111_1000) == 0b1111_0000) |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2527 { |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2528 d &= 0b0000_0111; |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2529 mixin(appendSixBits ~ |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2530 checkNextByte ~ appendSixBits ~ |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2531 checkNextByte ~ appendSixBits); |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2532 } |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2533 else |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2534 return false; |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2535 |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2536 assert(isTrailByte(*p)); |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2537 if (!isValidChar(d) || !isUniAlpha(d)) |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2538 return false; |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2539 // Only advance pointer if this is a Unicode alpha character. |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2540 this.p = p; |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2541 return true; |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2542 } |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2543 |
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2544 /// Decodes the next UTF-8 sequence. |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2545 dchar decodeUTF8() |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2546 { |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2547 assert(!isascii(*p), "check for ASCII char before calling decodeUTF8()."); |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2548 char* p = this.p; |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2549 dchar d = *p; |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2550 |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2551 ++p; // Move to second byte. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2552 // Error if second byte is not a trail byte. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2553 if (!isTrailByte(*p)) |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2554 goto Lerr2; |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2555 |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2556 // Check for overlong sequences. |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2557 switch (d) |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2558 { |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2559 case 0xE0, // 11100000 100xxxxx |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2560 0xF0, // 11110000 1000xxxx |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2561 0xF8, // 11111000 10000xxx |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2562 0xFC: // 11111100 100000xx |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2563 if ((*p & d) == 0x80) |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2564 goto Lerr; |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2565 default: |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2566 if ((d & 0xFE) == 0xC0) // 1100000x |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2567 goto Lerr; |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2568 } |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2569 |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2570 const char[] checkNextByte = "if (!isTrailByte(*++p))" |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2571 " goto Lerr2;"; |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2572 const char[] appendSixBits = "d = (d << 6) | *p & 0b0011_1111;"; |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2573 |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2574 // Decode |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2575 if ((d & 0b1110_0000) == 0b1100_0000) |
550
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2576 { // 110xxxxx 10xxxxxx |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2577 d &= 0b0001_1111; |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2578 mixin(appendSixBits); |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2579 } |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2580 else if ((d & 0b1111_0000) == 0b1110_0000) |
550
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2581 { // 1110xxxx 10xxxxxx 10xxxxxx |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2582 d &= 0b0000_1111; |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2583 mixin(appendSixBits ~ |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2584 checkNextByte ~ appendSixBits); |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2585 } |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2586 else if ((d & 0b1111_1000) == 0b1111_0000) |
550
082126d78f90
Fixed lexing of identifiers.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
538
diff
changeset
|
2587 { // 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2588 d &= 0b0000_0111; |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2589 mixin(appendSixBits ~ |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2590 checkNextByte ~ appendSixBits ~ |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2591 checkNextByte ~ appendSixBits); |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2592 } |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2593 else |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2594 // 5 and 6 byte UTF-8 sequences are not allowed yet. |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2595 // 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2596 // 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2597 goto Lerr; |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2598 |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2599 assert(isTrailByte(*p)); |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2600 |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2601 if (!isValidChar(d)) |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2602 { |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2603 Lerr: |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2604 // Three cases: |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2605 // *) the UTF-8 sequence was successfully decoded but the resulting |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2606 // character is invalid. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2607 // p points to last trail byte in the sequence. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2608 // *) the UTF-8 sequence is overlong. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2609 // p points to second byte in the sequence. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2610 // *) the UTF-8 sequence has more than 4 bytes or starts with |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2611 // a trail byte. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2612 // p points to second byte in the sequence. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2613 assert(isTrailByte(*p)); |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2614 // Move to next ASCII character or lead byte of a UTF-8 sequence. |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2615 while (p < (end-1) && isTrailByte(*p)) |
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2616 ++p; |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2617 --p; |
426
3f7790d3f9d6
Improved decodeUTF8().
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
425
diff
changeset
|
2618 assert(!isTrailByte(p[1])); |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2619 Lerr2: |
430
e6c759e151cd
Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
426
diff
changeset
|
2620 d = REPLACEMENT_CHAR; |
425
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2621 error(this.p, MID.InvalidUTF8Sequence); |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2622 } |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2623 |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2624 this.p = p; |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2625 return d; |
6bf936bf3356
Added own UTF-8 decoding function.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
424
diff
changeset
|
2626 } |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
2627 |
519 | 2628 static void encodeUTF8(ref char[] str, dchar d) |
48 | 2629 { |
430
e6c759e151cd
Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
426
diff
changeset
|
2630 assert(!isascii(d), "check for ASCII char before calling encodeUTF8()."); |
510
dd3ce87b3569
Added module dil.Unicode.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
507
diff
changeset
|
2631 assert(isValidChar(d), "check if character is valid before calling encodeUTF8()."); |
430
e6c759e151cd
Fixed a few things regarding encoding/decoding UTF-8 sequences.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
426
diff
changeset
|
2632 |
519 | 2633 char[6] b = void; |
48 | 2634 if (d < 0x800) |
2635 { | |
2636 b[0] = 0xC0 | (d >> 6); | |
2637 b[1] = 0x80 | (d & 0x3F); | |
2638 str ~= b[0..2]; | |
2639 } | |
2640 else if (d < 0x10000) | |
2641 { | |
2642 b[0] = 0xE0 | (d >> 12); | |
2643 b[1] = 0x80 | ((d >> 6) & 0x3F); | |
2644 b[2] = 0x80 | (d & 0x3F); | |
2645 str ~= b[0..3]; | |
2646 } | |
2647 else if (d < 0x200000) | |
2648 { | |
2649 b[0] = 0xF0 | (d >> 18); | |
2650 b[1] = 0x80 | ((d >> 12) & 0x3F); | |
2651 b[2] = 0x80 | ((d >> 6) & 0x3F); | |
2652 b[3] = 0x80 | (d & 0x3F); | |
2653 str ~= b[0..4]; | |
2654 } | |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
2655 /+ // There are no 5 and 6 byte UTF-8 sequences yet. |
48 | 2656 else if (d < 0x4000000) |
2657 { | |
2658 b[0] = 0xF8 | (d >> 24); | |
2659 b[1] = 0x80 | ((d >> 18) & 0x3F); | |
2660 b[2] = 0x80 | ((d >> 12) & 0x3F); | |
2661 b[3] = 0x80 | ((d >> 6) & 0x3F); | |
2662 b[4] = 0x80 | (d & 0x3F); | |
2663 str ~= b[0..5]; | |
2664 } | |
2665 else if (d < 0x80000000) | |
2666 { | |
2667 b[0] = 0xFC | (d >> 30); | |
2668 b[1] = 0x80 | ((d >> 24) & 0x3F); | |
2669 b[2] = 0x80 | ((d >> 18) & 0x3F); | |
2670 b[3] = 0x80 | ((d >> 12) & 0x3F); | |
2671 b[4] = 0x80 | ((d >> 6) & 0x3F); | |
2672 b[5] = 0x80 | (d & 0x3F); | |
2673 str ~= b[0..6]; | |
2674 } | |
424
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
2675 +/ |
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
2676 else |
bb3cb00feeb2
Applied some fixes to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
422
diff
changeset
|
2677 assert(0); |
48 | 2678 } |
5 | 2679 } |
39
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2680 |
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2681 unittest |
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2682 { |
391
33b566df6af4
Migrated project to Tango.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
390
diff
changeset
|
2683 Stdout("Testing Lexer.\n"); |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2684 struct Pair |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2685 { |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2686 char[] tokenText; |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2687 TOK kind; |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2688 } |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2689 static Pair[] pairs = [ |
494
9a7ca8c56e59
Refactored a few things in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
490
diff
changeset
|
2690 {"#!äöüß", TOK.Shebang}, {"\n", TOK.Newline}, |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2691 {"//çay", TOK.Comment}, {"\n", TOK.Newline}, |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2692 {"&", TOK.AndBinary}, |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2693 {"/*çağ*/", TOK.Comment}, {"&&", TOK.AndLogical}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2694 {"/+çak+/", TOK.Comment}, {"&=", TOK.AndAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2695 {">", TOK.Greater}, {"+", TOK.Plus}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2696 {">=", TOK.GreaterEqual}, {"++", TOK.PlusPlus}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2697 {">>", TOK.RShift}, {"+=", TOK.PlusAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2698 {">>=", TOK.RShiftAssign}, {"-", TOK.Minus}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2699 {">>>", TOK.URShift}, {"--", TOK.MinusMinus}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2700 {">>>=", TOK.URShiftAssign}, {"-=", TOK.MinusAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2701 {"<", TOK.Less}, {"=", TOK.Assign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2702 {"<=", TOK.LessEqual}, {"==", TOK.Equal}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2703 {"<>", TOK.LorG}, {"~", TOK.Tilde}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2704 {"<>=", TOK.LorEorG}, {"~=", TOK.CatAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2705 {"<<", TOK.LShift}, {"*", TOK.Mul}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2706 {"<<=", TOK.LShiftAssign}, {"*=", TOK.MulAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2707 {"!", TOK.Not}, {"/", TOK.Div}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2708 {"!=", TOK.NotEqual}, {"/=", TOK.DivAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2709 {"!<", TOK.UorGorE}, {"^", TOK.Xor}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2710 {"!>", TOK.UorLorE}, {"^=", TOK.XorAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2711 {"!<=", TOK.UorG}, {"%", TOK.Mod}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2712 {"!>=", TOK.UorL}, {"%=", TOK.ModAssign}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2713 {"!<>", TOK.UorE}, {"(", TOK.LParen}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2714 {"!<>=", TOK.Unordered}, {")", TOK.RParen}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2715 {".", TOK.Dot}, {"[", TOK.LBracket}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2716 {"..", TOK.Slice}, {"]", TOK.RBracket}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2717 {"...", TOK.Ellipses}, {"{", TOK.LBrace}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2718 {"|", TOK.OrBinary}, {"}", TOK.RBrace}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2719 {"||", TOK.OrLogical}, {":", TOK.Colon}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2720 {"|=", TOK.OrAssign}, {";", TOK.Semicolon}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2721 {"?", TOK.Question}, {",", TOK.Comma}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2722 {"$", TOK.Dollar}, {"cam", TOK.Identifier}, |
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2723 {"çay", TOK.Identifier}, {".0", TOK.Float64}, |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2724 {"0", TOK.Int32}, {"\n", TOK.Newline}, |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2725 {"\r", TOK.Newline}, {"\r\n", TOK.Newline}, |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2726 {"\u2028", TOK.Newline}, {"\u2029", TOK.Newline} |
40 | 2727 ]; |
39
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2728 |
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2729 char[] src; |
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2730 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2731 // Join all token texts into a single string. |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2732 foreach (i, pair; pairs) |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2733 if (pair.kind == TOK.Comment && pair.tokenText[1] == '/' || // Line comment. |
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2734 pair.kind == TOK.Shebang) |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2735 { |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2736 assert(pairs[i+1].kind == TOK.Newline); // Must be followed by a newline. |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2737 src ~= pair.tokenText; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2738 } |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2739 else |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2740 src ~= pair.tokenText ~ " "; |
39
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2741 |
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2742 auto lx = new Lexer(src, ""); |
350 | 2743 auto token = lx.getTokens(); |
39
69b940398d7b
- Added unittest to test correct parsing of operator tokens.
aziz
parents:
38
diff
changeset
|
2744 |
350 | 2745 uint i; |
2746 assert(token == lx.head); | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2747 assert(token.next.kind == TOK.Newline); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2748 token = token.next.next; |
350 | 2749 do |
2750 { | |
413
0fd78fdcb982
Added an alternative scan() method to class Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
412
diff
changeset
|
2751 assert(i < pairs.length); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2752 assert(token.srcText == pairs[i].tokenText, Format("Scanned '{0}' but expected '{1}'", token.srcText, pairs[i].tokenText)); |
350 | 2753 ++i; |
2754 token = token.next; | |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2755 } while (token.kind != TOK.EOF) |
41 | 2756 } |
55 | 2757 |
67
996065105910
- Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents:
66
diff
changeset
|
2758 unittest |
996065105910
- Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents:
66
diff
changeset
|
2759 { |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2760 Stdout("Testing method Lexer.peek()\n"); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2761 string sourceText = "unittest { }"; |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2762 auto lx = new Lexer(sourceText, null); |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2763 |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2764 auto next = lx.head; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2765 lx.peek(next); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2766 assert(next.kind == TOK.Newline); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2767 lx.peek(next); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2768 assert(next.kind == TOK.Unittest); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2769 lx.peek(next); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2770 assert(next.kind == TOK.LBrace); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2771 lx.peek(next); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2772 assert(next.kind == TOK.RBrace); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2773 lx.peek(next); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2774 assert(next.kind == TOK.EOF); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2775 |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2776 lx = new Lexer("", null); |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2777 next = lx.head; |
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2778 lx.peek(next); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2779 assert(next.kind == TOK.Newline); |
485
ea8c7459f1c4
Changed a lot of things in the Lexer.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
464
diff
changeset
|
2780 lx.peek(next); |
679
ff6971637f88
Renamed Token member type to kind.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
607
diff
changeset
|
2781 assert(next.kind == TOK.EOF); |
422
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2782 } |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2783 |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2784 unittest |
ad7977fe315a
Added support for column numbers in error messages.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
420
diff
changeset
|
2785 { |
67
996065105910
- Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents:
66
diff
changeset
|
2786 // Numbers unittest |
68
7eb83dd38901
- Simplified suffix rule and added a few more numbers to unittest.
aziz
parents:
67
diff
changeset
|
2787 // 0L 0ULi 0_L 0_UL 0x0U 0x0p2 0_Fi 0_e2 0_F 0_i |
7eb83dd38901
- Simplified suffix rule and added a few more numbers to unittest.
aziz
parents:
67
diff
changeset
|
2788 // 0u 0U 0uL 0UL 0L 0LU 0Lu |
7eb83dd38901
- Simplified suffix rule and added a few more numbers to unittest.
aziz
parents:
67
diff
changeset
|
2789 // 0Li 0f 0F 0fi 0Fi 0i |
7eb83dd38901
- Simplified suffix rule and added a few more numbers to unittest.
aziz
parents:
67
diff
changeset
|
2790 // 0b_1_LU 0b1000u |
7eb83dd38901
- Simplified suffix rule and added a few more numbers to unittest.
aziz
parents:
67
diff
changeset
|
2791 // 0x232Lu |
67
996065105910
- Fix: variadic arguments are local to the variadic function. Parameters are converted to an array of strings first and then passed to the constructor of class Problem.
aziz
parents:
66
diff
changeset
|
2792 } |