Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
| Comment: | Thoughts on content-addressible storage |
|---|---|
| Timelines: | family | ancestors | descendants | both | trunk |
| Files: | files | file ages | folders |
| SHA1: |
9ed1933bcebc686eb9981c1857e927cf |
| User & Date: | alaric 2013-06-27 15:42:55 |
Context
|
2013-07-16
| ||
| 20:47 | XENON added check-in: dd84df30e7 user: alaric tags: trunk | |
|
2013-06-27
| ||
| 15:42 | Thoughts on content-addressible storage check-in: 9ed1933bce user: alaric tags: trunk | |
|
2012-12-06
| ||
| 20:23 | More elaboration of CARBON check-in: f22fe263a5 user: alaric tags: trunk | |
Changes
Changes to README.wiki.
| ︙ | ︙ | |||
359 360 361 362 363 364 365 |
* Have a look in the WOLFRAM docs about the contents of the cluster
entity, and specify what fraction of it needs to be cached by
NITROGEN (which is also the subset known to a TUNGSTEN-less
node). What can we do without? How much RAM will it take? This
affects the scaling limits of a cluster with simple nodes.
| | > > | | > > | > > > > > > > | | | | > | | 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 |
* Have a look in the WOLFRAM docs about the contents of the cluster
entity, and specify what fraction of it needs to be cached by
NITROGEN (which is also the subset known to a TUNGSTEN-less
node). What can we do without? How much RAM will it take? This
affects the scaling limits of a cluster with simple nodes.
* Document CARBON/MERCURY interfaces to cluster, volume, and node
entities. Think about the services they should offer to
administrators as well as to users.
* Give TUNGSTEN a content-addressed archive storage mode, and think
about how to use it as part of the backup model. Perhaps groups
of assertions in TUNGSTEN knowledge bases that do not seem to
change can be considered eligible for compaction into the CAS for
efficiency reasons, or perhaps entities should have explicit
access to immutable CARBON knowledge bases (initialised from a
mutable KB and identified by hash thereafter). If published KB
sections are frozen to the CAS somehow, then the very byte
sequence stored in the CAS can be shipped direct to clients via
MERCURY, including the potential bittorrent-style protocol
extension, which would offer significant reduction in the layers
of software required to service such requests, while still having
a nice programming model. This could also be a basis for
document-style entities to be able to provide past versions of
themselves via the persona field. Voila, version control... How
do we provide the semantics and workflow of a DVCS, though? An
IODINE interface provided by document entities to pull and push
changes from a clone entity?
* Put Disqus or something on this site so people can comment and
ask questions.
* Create Docbook specifications
* Create an index of symbols defined in the specs under
|
| ︙ | ︙ | |||
392 393 394 395 396 397 398 |
CARBON and TUNGSTEN writings to not assume this as a default
namespace prefix. There is no default namespace prefix in those
cases, so using a default-namespace name without declaring one is
illegal.
<h2>Further research required</h2>
| | | | > | | | | | | | | | | | | | | | | | | | 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 |
CARBON and TUNGSTEN writings to not assume this as a default
namespace prefix. There is no default namespace prefix in those
cases, so using a default-namespace name without declaring one is
illegal.
<h2>Further research required</h2>
* Design a logging framework (PLATINUM). I want all logs integrated
at one point, then distributed as needed, rather than ending up
correlating disparate logs. I want logs structured with enough
metadata to allow finding relationships between events so log
entries can be "folded up" to hide fine detail (eg, a MERCURY
incoming request will start off being logged by IRIDIUM then lead
to a LITHIUM invocation which will lead to CHROME activity and
then user code calling into WOLFRAM, TUNGSTEN and MERCURY to do
stuff, then back to MERCURY to IRIDIUM to return a result, then
WOLFRAM to commit the TUNGSTEN state transaction, etc - but it
should be foldable up into a single "MERCURY invocation" summary
event. We don't need CEP systems for this, just threading of
event IDs down through the system as part of the dynamic
execution context and appropriate importance levels. Log events
should be represented in CARBON, so using these links shouldn't
be hard. Also, node-specific events need to be fed into the
HYDROGEN console system for diagnosis, and a storage system of
some kind chosen for log events that need recording (with
per-handler, per-entity, per-volume, per-node and cluster-wide
log level settings to control storage utilisation, and the
ability to remove levels of detail in older logs as they are
expired). Where are logs stored? Within entities, or as a
special case with direct TUNGSTEN storage? What gets replicated
when? Do we log events up to level N to the node only, and events
up to some (lower) level M get replicated in real time because
they're important and we need them to not be tamperable if a node
is compromised, and the level N logs on the node get summarised
to level X and then replicated every day for archival purposes?
And who can see the logs? A way is needed for logs about a given
entity to be made available to the administrators of that entity
(see also the debug mode tracing mechanism in LITHIUM/ARGON,
which need to boil down to this). This is important and major
enough to need an element name, I feel!
* Should we allow for N-dimensional arrays as well as just vectors
in IRON? They would allow for better predictive coding for sensor
data such as images, as we'd be able to take into account pixels
above as well as to the left of the next one (and the
generalisation into higher dimensions thereof). If so, update the
CARBON page's image storage example.
|
| ︙ | ︙ |