# HG changeset patch # User Aziz K?ksal # Date 1201791490 -3600 # Node ID b6c6baa4126770154aa57600bccfaf8bdcead80f # Parent 140469ecb90e04e614326336880b513da6de01ce Moved character properties table to dil.lexer.Funcs. diff -r 140469ecb90e -r b6c6baa41267 trunk/src/dil/lexer/Funcs.d --- a/trunk/src/dil/lexer/Funcs.d Thu Jan 31 01:10:30 2008 +0100 +++ b/trunk/src/dil/lexer/Funcs.d Thu Jan 31 15:58:10 2008 +0100 @@ -80,3 +80,92 @@ } return 0; } + +/// ASCII character properties table. +static const int ptable[256] = [ + 0, 0, 0, 0, 0, 0, 0, 0, 0,32, 0,32,32, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +32, 0, 0x2200, 0, 0, 0, 0, 0x2700, 0, 0, 0, 0, 0, 0, 0, 0, + 7, 7, 7, 7, 7, 7, 7, 7, 6, 6, 0, 0, 0, 0, 0, 0x3f00, + 0,12,12,12,12,12,12, 8, 8, 8, 8, 8, 8, 8, 8, 8, + 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0, 0x5c00, 0, 0,16, + 0, 0x70c, 0x80c,12,12,12, 0xc0c, 8, 8, 8, 8, 8, 8, 8, 0xa08, 8, + 8, 8, 0xd08, 8, 0x908, 8, 0xb08, 8, 8, 8, 8, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 +]; + +enum CProperty +{ + Octal = 1, + Digit = 1<<1, + Hex = 1<<2, + Alpha = 1<<3, + Underscore = 1<<4, + Whitespace = 1<<5 +} + +const uint EVMask = 0xFF00; // Bit mask for escape value + +private alias CProperty CP; +int isoctal(char c) { return ptable[c] & CP.Octal; } +int isdigit(char c) { return ptable[c] & CP.Digit; } +int ishexad(char c) { return ptable[c] & CP.Hex; } +int isalpha(char c) { return ptable[c] & CP.Alpha; } +int isalnum(char c) { return ptable[c] & (CP.Alpha | CP.Digit); } +int isidbeg(char c) { return ptable[c] & (CP.Alpha | CP.Underscore); } +int isident(char c) { return ptable[c] & (CP.Alpha | CP.Underscore | CP.Digit); } +int isspace(char c) { return ptable[c] & CP.Whitespace; } +int char2ev(char c) { return ptable[c] >> 8; /*(ptable[c] & EVMask) >> 8;*/ } +int isascii(uint c) { return c < 128; } + +version(gen_ptable) +static this() +{ + alias ptable p; + assert(p.length == 256); + // Initialize character properties table. + for (int i; i < p.length; ++i) + { + p[i] = 0; // Reset + if ('0' <= i && i <= '7') + p[i] |= CP.Octal; + if ('0' <= i && i <= '9') + p[i] |= CP.Digit | CP.Hex; + if ('a' <= i && i <= 'f' || 'A' <= i && i <= 'F') + p[i] |= CP.Hex; + if ('a' <= i && i <= 'z' || 'A' <= i && i <= 'Z') + p[i] |= CP.Alpha; + if (i == '_') + p[i] |= CP.Underscore; + if (i == ' ' || i == '\t' || i == '\v' || i == '\f') + p[i] |= CP.Whitespace; + } + // Store escape sequence values in second byte. + assert(CProperty.max <= ubyte.max, "character property flags and escape value byte overlap."); + p['\''] |= 39 << 8; + p['"'] |= 34 << 8; + p['?'] |= 63 << 8; + p['\\'] |= 92 << 8; + p['a'] |= 7 << 8; + p['b'] |= 8 << 8; + p['f'] |= 12 << 8; + p['n'] |= 10 << 8; + p['r'] |= 13 << 8; + p['t'] |= 9 << 8; + p['v'] |= 11 << 8; + // Print a formatted array literal. + char[] array = "[\n"; + foreach (i, c; ptable) + { + array ~= Format((c>255?" 0x{0:x},":"{0,2},"), c) ~ (((i+1) % 16) ? "":"\n"); + } + array[$-2..$] = "\n]"; + Stdout(array).newline; +} diff -r 140469ecb90e -r b6c6baa41267 trunk/src/dil/lexer/Lexer.d --- a/trunk/src/dil/lexer/Lexer.d Thu Jan 31 01:10:30 2008 +0100 +++ b/trunk/src/dil/lexer/Lexer.d Thu Jan 31 15:58:10 2008 +0100 @@ -2784,92 +2784,3 @@ // 0b_1_LU 0b1000u // 0x232Lu } - -/// ASCII character properties table. -static const int ptable[256] = [ - 0, 0, 0, 0, 0, 0, 0, 0, 0,32, 0,32,32, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -32, 0, 0x2200, 0, 0, 0, 0, 0x2700, 0, 0, 0, 0, 0, 0, 0, 0, - 7, 7, 7, 7, 7, 7, 7, 7, 6, 6, 0, 0, 0, 0, 0, 0x3f00, - 0,12,12,12,12,12,12, 8, 8, 8, 8, 8, 8, 8, 8, 8, - 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0, 0x5c00, 0, 0,16, - 0, 0x70c, 0x80c,12,12,12, 0xc0c, 8, 8, 8, 8, 8, 8, 8, 0xa08, 8, - 8, 8, 0xd08, 8, 0x908, 8, 0xb08, 8, 8, 8, 8, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 -]; - -enum CProperty -{ - Octal = 1, - Digit = 1<<1, - Hex = 1<<2, - Alpha = 1<<3, - Underscore = 1<<4, - Whitespace = 1<<5 -} - -const uint EVMask = 0xFF00; // Bit mask for escape value - -private alias CProperty CP; -int isoctal(char c) { return ptable[c] & CP.Octal; } -int isdigit(char c) { return ptable[c] & CP.Digit; } -int ishexad(char c) { return ptable[c] & CP.Hex; } -int isalpha(char c) { return ptable[c] & CP.Alpha; } -int isalnum(char c) { return ptable[c] & (CP.Alpha | CP.Digit); } -int isidbeg(char c) { return ptable[c] & (CP.Alpha | CP.Underscore); } -int isident(char c) { return ptable[c] & (CP.Alpha | CP.Underscore | CP.Digit); } -int isspace(char c) { return ptable[c] & CP.Whitespace; } -int char2ev(char c) { return ptable[c] >> 8; /*(ptable[c] & EVMask) >> 8;*/ } -int isascii(uint c) { return c < 128; } - -version(gen_ptable) -static this() -{ - alias ptable p; - assert(p.length == 256); - // Initialize character properties table. - for (int i; i < p.length; ++i) - { - p[i] = 0; // Reset - if ('0' <= i && i <= '7') - p[i] |= CP.Octal; - if ('0' <= i && i <= '9') - p[i] |= CP.Digit; - if (isdigit(i) || 'a' <= i && i <= 'f' || 'A' <= i && i <= 'F') - p[i] |= CP.Hex; - if ('a' <= i && i <= 'z' || 'A' <= i && i <= 'Z') - p[i] |= CP.Alpha; - if (i == '_') - p[i] |= CP.Underscore; - if (i == ' ' || i == '\t' || i == '\v' || i == '\f') - p[i] |= CP.Whitespace; - } - // Store escape sequence values in second byte. - assert(CProperty.max <= ubyte.max, "character property flags and escape value byte overlap."); - p['\''] |= 39 << 8; - p['"'] |= 34 << 8; - p['?'] |= 63 << 8; - p['\\'] |= 92 << 8; - p['a'] |= 7 << 8; - p['b'] |= 8 << 8; - p['f'] |= 12 << 8; - p['n'] |= 10 << 8; - p['r'] |= 13 << 8; - p['t'] |= 9 << 8; - p['v'] |= 11 << 8; - // Print a formatted array literal. - char[] array = "[\n"; - foreach (i, c; ptable) - { - array ~= Format((c>255?" 0x{0:x},":"{0,2},"), c) ~ (((i+1) % 16) ? "":"\n"); - } - array[$-2..$] = "\n]"; - Stdout(array).newline; -}