Mercurial > projects > doodle

comparison nobuild/doc/doodle.rst @ 138:a1c2b56cb44d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Beginnings of rst doc with embedded plantuml
author David Bryant <bagnose@gmail.com>
date Sun, 30 Sep 2012 15:25:53 +0930
parents
children
comparison
equal deleted inserted replaced
137:9a9dcae45e08 138:a1c2b56cb44d
1 ==================
2 Doodle Development
3 ==================
4
5 Introduction
6 ============
7
8 Objectives
9 ----------
10
11 Build Tool
12 ----------
13
14 Directory Organisation
15 ----------------------
16
17 Crucial Design Aspects
18 ======================
19
20 Tool Stack
21 ----------
22
23 Interaction, event handling, etc.
24
25 Undo Framework
26 --------------
27
28 Drawing Abstraction
29 -------------------
30
31 Detailed Component Descriptions
32 ===============================
33
34 CUT !!!
35
36 Scope
37 =====
38
39 Overview
40 ========
41
42 Definitions
43 -----------
44
45 Blocklet
46
47 Variable length block of data produced by breaking blobs on boundaries
48 such that identical blocklets are likely to occur in other blobs.
49 Internal to blockpool.
50
51 Blob
52
53 Arbitrary user data blocks stored/retrieved in blockpool.
54
55 Tag
56 A small identifier that refers to a blob within a blockpool.
57
58 Hash
59 A small identifier that refers to a blocklet within a blockpool.
60 Internal to blockpool.
61
62 BFST-lib
63 A shared-library (and associated header files) for invoking
64 BFST functions from another program.
65
66 BFST-CLI
67 The blocklets filestore executable.
68
69 BFST-API
70 A set of functions provided by BFST-lib for directly
71 accessing a blockpool or issuing commands to a BFST server.
72
73 Blockpool
74 A blockpool on disk.
75
76 Purpose
77 --------
78
79 BFST is a store/retrieve technology that increases effective data capacity by
80 eliminating redundant data. BFST presents a simple external API:
81
82 - blob -> tag: Store a blob (arbitrary block of data) in the blockpool and
83 receive a tag (small unique identifier) associated with the blob.
84
85 - tag -> blob: Retrieve a blob from the blockpool by providing the tag
86 associated with the blob.
87
88 Also supported: replaction between blockpools.
89
90 Higher level services can be built on top of blockpool, for example, a
91 DXi presents a file-server that uses BFST as a deduplication backend.
92
93 Operational Modes
94 -----------------
95
96 Blockpool operations can be performed via:
97
98 API calls:
99 Calling functions expored by libbfst.so.
100 (Local and remote operations.)
101
102 CLI execution:
103 Executing blockpool with chosen arguments.
104 (Local, remote and server launching.)
105
106 The blockpool executable can be launched in the following modes:
107
108 Local
109 Perform an operation on a local blockpool.
110 (Note deployed in this manner)
111
112 Client / Remote
113
114 Server
115
116 A Blockpool server can be conversed with in two ways:
117
118 - CLI mode
119
120 - API mode
121
122 Implementation
123 --------------
124
125 Repository Layout
126 -----------------
127
128 Simplified layout::
129
130 bfst
131 |
132 +--lib_rocksoft_c
133 |
134 +--libfilter Stream filtering library, eg find tar boundary.
135 |
136 +--blockpool
137 |
138 +--perl_mod ADLQuantum perl modules.
139 |
140 +--test
141
142 Supported Platforms
143 -------------------
144
145 Building
146 --------
147
148 Linux
149 `````
150
151 Running
152 -------
153
154 Testing
155 -------
156
157 - self tests (./blockpool/cli/blockpool selftest default)
158
159 - perl test harness ./test/scripts/test.sh
160
161 Detailed Description
162 ====================
163
164 Component: lib_rocksoft_c
165 -------------------------
166
167 Layout::
168
169 lib_rocksoft_c
170 |
171 +--low
172 |
173 +--ADLQuantum
174 |
175 +--test Self tests (unit tests)
176 |
177 +--external 3rd-party software
178
179 Component: libfilter
180 --------------------
181
182 Component: blockpool
183 --------------------
184
185 Layout::
186
187 blockpool
188 |
189 +--memory_budgets
190 |
191 +--api
192 |
193 +--cli
194 |
195 +--lib
196 | |
197 | +--test Self tests (unit tests)
198 | |
199 | +--ADLQuantum More perl modules
200 |
201 +--scripts
202 |
203 +--sdk
204 |
205 +--docs
206
207
208
209 Important:
210
211 - In a DXi, blockpool runs as in server mode.
212
213 - BPW is a client (one of many) it maintains a preconfigured
214 number of connections as a pool. These are used as required.
215
216 - shared memory segment runs from end-point (nfs/cifs/vtl/ost),
217 thru BPW and to bfst server.
218
219 - pcache is a portion of the shared segment that is used to communicated
220 between endpoints and BPW. BFST doesn't see pcache.
221
222 - BPW links the BFST shared library.
223
224 - When a new blob is stored in BFST, it is converted into blocklets,
225 if the blocklet matches then its ref-count is incremented, otherwise
226 a new blocklet is added. At the end of the conversion, the list of
227 hashes is hashed to form the blob tag.
228 If it turns out a blob with that tag already exists then the ref-count
229 of that tag is incremented, but the blocklets' ref-counts are NOT
230 decremented (to save cost). Somehow (I can't think how) everything
231 works out ok when the blobs are unreferenced.
232
233 - Note, when replication is being used a blob's ref-count can be incremented
234 without incrementing the ref-counts of the corresponding blocklets.
235 How in the hell is this discrepancy reconciled without leaking blocklets
236 or deleting blocklets in use.
237
238 Appendix
239 ========
240
241 Misc
242 ----
243
244 .. uml:: images/class-hierarchy
245
246 package "BPW" #ffffff {
247 class File
248 class TagRef
249 }
250
251 package "BFST" #ffffff {
252 class Blob {
253 byte[64] tag
254 int ref-count
255 }
256
257 class HashRef
258
259 class Blocklet {
260 byte[64] hash
261 int ref-count
262 Flags flags
263 byte[] data
264 }
265 }
266
267 File "1" *- "1..*" TagRef
268 TagRef "1..*" *-- "1" Blob
269
270 Blob "1" *- "1..*" HashRef
271 HashRef "1..*" *- "1" Blocklet
272
273 hide methods
274
275 - store: blob gets converted into blocklets, matched blocklets have
276 ref-count incremented, new blocklets added. At the end we calculate
277 the tag from the hash of hashes. Then if the tag matches an existing
278 blob we increment the blob's ref-count AND the blob's blocklet-ref-count.
279
280 - replicate: we have the tag of the blob, if there is a match then increment
281 the blob's ref-count but NOT the bloc's blocklet-ref-count.
282
283 Documentation Resources
284 -----------------------
285
286 - make USE_PLANTUML=true doc ; ${BROWSER} index.html
287
288 - ${BFST_ROOT}/blockpool/docs, ${BFST_ROOT}/libfilter/docs,
289 ${BFST_ROOT}/blockpool/api/docs, ${BFST_ROOT}/blockpool/cli/docs
290
291 - Wiki links
292
293 - Misc
294
295 - http://wiki.adl.quantum.com/index.php/Tips_and_Tricks
296
297 - Building on Linux:
298
299 - http://wiki.adl.quantum.com/index.php/Compiling_Blockpool_On_Linux
300
301 - http://wiki.adl.quantum.com/index.php/BuildInfrastructure
302
303 - `Document repository
304 <http://wiki.adl.quantum.com/index.php/Document_Repository>`_
305
306 - `C Coding Standard <http://inside.adl.quantum.com/nlee/cstan/>`_