ARGON
Check-in [5cfd191570]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Noted that one can schedule SINGLE jobs as well as job generators in HELIUM and WOLFRAM.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 5cfd191570278c1089cbe2fa4cb16e42b5d3c195
User & Date: alaric 2012-07-23 16:31:38
Context
2012-07-26
08:36
wolfram: FIXME about lightweight job generation interface check-in: e6c37344e9 user: alaric tags: trunk
2012-07-23
16:31
Noted that one can schedule SINGLE jobs as well as job generators in HELIUM and WOLFRAM. check-in: 5cfd191570 user: alaric tags: trunk
2012-07-13
15:06
Fixed METADATA.rdf check-in: 0439d2fa47 user: alaric tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to intro/helium.wiki.

65
66
67
68
69
70
71
72


73

74
75
76
77
78
79
80

81
82
83
84
85
86
87
create and splitting the workload between them, however, such
libraries are encouraged to instead register a "job generator"
callback with HELIUM, which will be invoked whenever there is no work
available at that priority level or higher, and which will return
bounded jobs until there are none left; that way, HELIUM will
automatically split the jobs over as many processors as are available,
taking into account workloads at the same or higher priority, and how
CPU-bound the jobs are, automatically and efficiently.




In addition, WOLFRAM will provide facilities to entity code
to request that algorithms be distributed across the cluster. Again, a
job generator is provided, which produces jobs on demand; but WOLFRAM
jobs are specified as an entity handler to invoke with LITHIUM and a
job object to pass to it. WOLFRAM communicates with available nodes to
register job generators with HELIUM, that will in turn call the
distributed job generator and run the jobs in parallel via LITHIUM.


Finally, low-priority "idle job generators" may be configured
cluster-wide or for individual nodes, to try to find uses for spare
CPU power for grid computing applications.

The CPU scheduling priorities can also be used to schedule access to
disk bandwidth (to make higher-priority commits more rapid) and







|
>
>

>
|
|
|
|
|
|
|
>







65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
create and splitting the workload between them, however, such
libraries are encouraged to instead register a "job generator"
callback with HELIUM, which will be invoked whenever there is no work
available at that priority level or higher, and which will return
bounded jobs until there are none left; that way, HELIUM will
automatically split the jobs over as many processors as are available,
taking into account workloads at the same or higher priority, and how
CPU-bound the jobs are, automatically and efficiently. And, of course,
it'll be possible to submit a single job to do, which goes into the
same pool of jobs.

However, user code will not directly use HELIUM's scheduling
interface; WOLFRAM will provide facilities to entity code to request
that algorithms be distributed across the cluster. Again, a job
generator is provided, which produces jobs on demand, or a single job;
but WOLFRAM jobs are specified as an entity handler to invoke with
LITHIUM and a job object to pass to it. WOLFRAM communicates with
available nodes to register job generators with HELIUM, that will in
turn call the distributed job generator and run the jobs in parallel
via LITHIUM.

Finally, low-priority "idle job generators" may be configured
cluster-wide or for individual nodes, to try to find uses for spare
CPU power for grid computing applications.

The CPU scheduling priorities can also be used to schedule access to
disk bandwidth (to make higher-priority commits more rapid) and

Changes to intro/wolfram.wiki.

322
323
324
325
326
327
328
329




330
331
332
333
334
335
336
WOLFRAM will install a [./helium.wiki|HELIUM] job generator callback
at a specified priority level. This callback will invoke the specified
LITHIUM handler in the entity to obtain the ID of a second LITHIUM
handler and arguments that represent the actual task to perform. This
will continue until the original handler stops providing jobs; when
all of the jobs have completed, the distributed job as a whole is
complete. The job generator system basically implements a lazy list of
jobs to do, allowing it to be computed on the fly.





<h1>Distributed caching</h1>

WOLFRAM also provides a distributed cache mechanism.

There are several caches in the system. As well as one cache created
implicitly per node, which is stored only on that node, and a global







|
>
>
>
>







322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
WOLFRAM will install a [./helium.wiki|HELIUM] job generator callback
at a specified priority level. This callback will invoke the specified
LITHIUM handler in the entity to obtain the ID of a second LITHIUM
handler and arguments that represent the actual task to perform. This
will continue until the original handler stops providing jobs; when
all of the jobs have completed, the distributed job as a whole is
complete. The job generator system basically implements a lazy list of
jobs to do, allowing it to be computed on the fly. And for
parallelisable tasks that aren't generated in bulk, it will be
possible to submit a single job to be run, as a LITHIUM handler ID and
arguments, which may be distributed to a lightly-loaded node in the
cluster to be run if the local node is busy.

<h1>Distributed caching</h1>

WOLFRAM also provides a distributed cache mechanism.

There are several caches in the system. As well as one cache created
implicitly per node, which is stored only on that node, and a global