Mercurial > projects > dil

annotate trunk/src/dil/lexer/Funcs.d @ 715:b6c6baa41267

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Moved character properties table to dil.lexer.Funcs.
author Aziz K?ksal <aziz.koeksal@gmail.com>
date Thu, 31 Jan 2008 15:58:10 +0100
parents 9e811db780a6
children 5e3ef1b2011c
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
1 /++
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
2 Author: Aziz Köksal
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
3 License: GPL3
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
4 +/
577
9e811db780a6 Moved LexerFuncs.d to package 'lexer'.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 533
diff changeset
5 module dil.lexer.Funcs;
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
6
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
7 const char[3] LS = \u2028; /// Line separator.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
8 const char[3] PS = \u2029; /// Paragraph separator.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
9 const dchar LSd = 0x2028;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
10 const dchar PSd = 0x2029;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
11 static assert(LS[0] == PS[0] && LS[1] == PS[1]);
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
12
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
13 const uint _Z_ = 26; /// Control+Z
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
14
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
15 /// Returns true if d is a Unicode line or paragraph separator.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
16 bool isUnicodeNewlineChar(dchar d)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
17 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
18 return d == LSd || d == PSd;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
19 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
20
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
21 /// Returns true if p points to a line or paragraph separator.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
22 bool isUnicodeNewline(char* p)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
23 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
24 return *p == LS[0] && p[1] == LS[1] && (p[2] == LS[2] || p[2] == PS[2]);
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
25 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
26
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
27 /++
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
28 Returns true if p points to the start of a Newline.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
29 Newline: \n | \r | \r\n | LS | PS
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
30 +/
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
31 bool isNewline(char* p)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
32 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
33 return *p == '\n' || *p == '\r' || isUnicodeNewline(p);
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
34 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
35
533
2a8d0ed0d71e Improved error reporting in dil.Converter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
36 /// Returns if c is a Newline character.
2a8d0ed0d71e Improved error reporting in dil.Converter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
37 bool isNewline(dchar c)
2a8d0ed0d71e Improved error reporting in dil.Converter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
38 {
2a8d0ed0d71e Improved error reporting in dil.Converter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
39 return c == '\n' || c == '\r' || isUnicodeNewlineChar(c);
2a8d0ed0d71e Improved error reporting in dil.Converter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
40 }
2a8d0ed0d71e Improved error reporting in dil.Converter.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 496
diff changeset
41
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
42 /++
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
43 Returns true if p points to an EOF character.
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
44 EOF: 0 | _Z_
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
45 +/
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
46 bool isEOF(dchar c)
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
47 {
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
48 return c == 0 || c == _Z_;
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
49 }
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
50
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
51 /++
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
52 Returns true if p points to the first character of an EndOfLine.
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
53 EndOfLine: Newline | EOF
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
54 +/
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
55 bool isEndOfLine(char* p)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
56 {
496
5a607597dc22 Improved error recovery in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 494
diff changeset
57 return isNewline(p) || isEOF(*p);
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
58 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
59
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
60 /++
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
61 Scans a Newline and sets p one character past it.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
62 Returns '\n' if scanned or 0 otherwise.
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
63 +/
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
64 dchar scanNewline(ref char* p)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
65 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
66 switch (*p)
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
67 {
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
68 case '\r':
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
69 if (p[1] == '\n')
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
70 ++p;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
71 case '\n':
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
72 ++p;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
73 return '\n';
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
74 default:
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
75 if (isUnicodeNewline(p))
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
76 {
494
9a7ca8c56e59 Refactored a few things in the Parser.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 490
diff changeset
77 p += 3;
490
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
78 return '\n';
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
79 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
80 }
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
81 return 0;
47be6bfe39cd Refactored code and added new modules.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents:
diff changeset
82 }
715
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
83
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
84 /// ASCII character properties table.
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
85 static const int ptable[256] = [
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
86 0, 0, 0, 0, 0, 0, 0, 0, 0,32, 0,32,32, 0, 0, 0,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
87 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
88 32, 0, 0x2200, 0, 0, 0, 0, 0x2700, 0, 0, 0, 0, 0, 0, 0, 0,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
89 7, 7, 7, 7, 7, 7, 7, 7, 6, 6, 0, 0, 0, 0, 0, 0x3f00,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
90 0,12,12,12,12,12,12, 8, 8, 8, 8, 8, 8, 8, 8, 8,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
91 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0, 0x5c00, 0, 0,16,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
92 0, 0x70c, 0x80c,12,12,12, 0xc0c, 8, 8, 8, 8, 8, 8, 8, 0xa08, 8,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
93 8, 8, 0xd08, 8, 0x908, 8, 0xb08, 8, 8, 8, 8, 0, 0, 0, 0, 0,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
94 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
95 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
96 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
97 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
98 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
99 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
100 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
101 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
102 ];
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
103
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
104 enum CProperty
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
105 {
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
106 Octal = 1,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
107 Digit = 1<<1,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
108 Hex = 1<<2,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
109 Alpha = 1<<3,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
110 Underscore = 1<<4,
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
111 Whitespace = 1<<5
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
112 }
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
113
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
114 const uint EVMask = 0xFF00; // Bit mask for escape value
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
115
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
116 private alias CProperty CP;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
117 int isoctal(char c) { return ptable[c] & CP.Octal; }
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
118 int isdigit(char c) { return ptable[c] & CP.Digit; }
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
119 int ishexad(char c) { return ptable[c] & CP.Hex; }
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
120 int isalpha(char c) { return ptable[c] & CP.Alpha; }
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
121 int isalnum(char c) { return ptable[c] & (CP.Alpha | CP.Digit); }
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
122 int isidbeg(char c) { return ptable[c] & (CP.Alpha | CP.Underscore); }
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
123 int isident(char c) { return ptable[c] & (CP.Alpha | CP.Underscore | CP.Digit); }
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
124 int isspace(char c) { return ptable[c] & CP.Whitespace; }
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
125 int char2ev(char c) { return ptable[c] >> 8; /*(ptable[c] & EVMask) >> 8;*/ }
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
126 int isascii(uint c) { return c < 128; }
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
127
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
128 version(gen_ptable)
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
129 static this()
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
130 {
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
131 alias ptable p;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
132 assert(p.length == 256);
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
133 // Initialize character properties table.
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
134 for (int i; i < p.length; ++i)
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
135 {
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
136 p[i] = 0; // Reset
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
137 if ('0' <= i && i <= '7')
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
138 p[i] |= CP.Octal;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
139 if ('0' <= i && i <= '9')
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
140 p[i] |= CP.Digit | CP.Hex;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
141 if ('a' <= i && i <= 'f' || 'A' <= i && i <= 'F')
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
142 p[i] |= CP.Hex;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
143 if ('a' <= i && i <= 'z' || 'A' <= i && i <= 'Z')
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
144 p[i] |= CP.Alpha;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
145 if (i == '_')
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
146 p[i] |= CP.Underscore;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
147 if (i == ' ' || i == '\t' || i == '\v' || i == '\f')
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
148 p[i] |= CP.Whitespace;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
149 }
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
150 // Store escape sequence values in second byte.
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
151 assert(CProperty.max <= ubyte.max, "character property flags and escape value byte overlap.");
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
152 p['\''] |= 39 << 8;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
153 p['"'] |= 34 << 8;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
154 p['?'] |= 63 << 8;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
155 p['\\'] |= 92 << 8;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
156 p['a'] |= 7 << 8;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
157 p['b'] |= 8 << 8;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
158 p['f'] |= 12 << 8;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
159 p['n'] |= 10 << 8;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
160 p['r'] |= 13 << 8;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
161 p['t'] |= 9 << 8;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
162 p['v'] |= 11 << 8;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
163 // Print a formatted array literal.
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
164 char[] array = "[\n";
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
165 foreach (i, c; ptable)
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
166 {
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
167 array ~= Format((c>255?" 0x{0:x},":"{0,2},"), c) ~ (((i+1) % 16) ? "":"\n");
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
168 }
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
169 array[$-2..$] = "\n]";
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
170 Stdout(array).newline;
b6c6baa41267 Moved character properties table to dil.lexer.Funcs.
Aziz K?ksal <aziz.koeksal@gmail.com>
parents: 577
diff changeset
171 }