Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
Comment: | Noted that one can schedule SINGLE jobs as well as job generators in HELIUM and WOLFRAM. |
---|---|
Timelines: | family | ancestors | descendants | both | trunk |
Files: | files | file ages | folders |
SHA1: |
5cfd191570278c1089cbe2fa4cb16e42 |
User & Date: | alaric 2012-07-23 16:31:38 |
Context
2012-07-26
| ||
08:36 | wolfram: FIXME about lightweight job generation interface check-in: e6c37344e9 user: alaric tags: trunk | |
2012-07-23
| ||
16:31 | Noted that one can schedule SINGLE jobs as well as job generators in HELIUM and WOLFRAM. check-in: 5cfd191570 user: alaric tags: trunk | |
2012-07-13
| ||
15:06 | Fixed METADATA.rdf check-in: 0439d2fa47 user: alaric tags: trunk | |
Changes
Changes to intro/helium.wiki.
︙ | ︙ | |||
65 66 67 68 69 70 71 | create and splitting the workload between them, however, such libraries are encouraged to instead register a "job generator" callback with HELIUM, which will be invoked whenever there is no work available at that priority level or higher, and which will return bounded jobs until there are none left; that way, HELIUM will automatically split the jobs over as many processors as are available, taking into account workloads at the same or higher priority, and how | | > > > | | | | | | | > | 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 | create and splitting the workload between them, however, such libraries are encouraged to instead register a "job generator" callback with HELIUM, which will be invoked whenever there is no work available at that priority level or higher, and which will return bounded jobs until there are none left; that way, HELIUM will automatically split the jobs over as many processors as are available, taking into account workloads at the same or higher priority, and how CPU-bound the jobs are, automatically and efficiently. And, of course, it'll be possible to submit a single job to do, which goes into the same pool of jobs. However, user code will not directly use HELIUM's scheduling interface; WOLFRAM will provide facilities to entity code to request that algorithms be distributed across the cluster. Again, a job generator is provided, which produces jobs on demand, or a single job; but WOLFRAM jobs are specified as an entity handler to invoke with LITHIUM and a job object to pass to it. WOLFRAM communicates with available nodes to register job generators with HELIUM, that will in turn call the distributed job generator and run the jobs in parallel via LITHIUM. Finally, low-priority "idle job generators" may be configured cluster-wide or for individual nodes, to try to find uses for spare CPU power for grid computing applications. The CPU scheduling priorities can also be used to schedule access to disk bandwidth (to make higher-priority commits more rapid) and |
︙ | ︙ |
Changes to intro/wolfram.wiki.
︙ | ︙ | |||
322 323 324 325 326 327 328 | WOLFRAM will install a [./helium.wiki|HELIUM] job generator callback at a specified priority level. This callback will invoke the specified LITHIUM handler in the entity to obtain the ID of a second LITHIUM handler and arguments that represent the actual task to perform. This will continue until the original handler stops providing jobs; when all of the jobs have completed, the distributed job as a whole is complete. The job generator system basically implements a lazy list of | | > > > > | 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 | WOLFRAM will install a [./helium.wiki|HELIUM] job generator callback at a specified priority level. This callback will invoke the specified LITHIUM handler in the entity to obtain the ID of a second LITHIUM handler and arguments that represent the actual task to perform. This will continue until the original handler stops providing jobs; when all of the jobs have completed, the distributed job as a whole is complete. The job generator system basically implements a lazy list of jobs to do, allowing it to be computed on the fly. And for parallelisable tasks that aren't generated in bulk, it will be possible to submit a single job to be run, as a LITHIUM handler ID and arguments, which may be distributed to a lightly-loaded node in the cluster to be run if the local node is busy. <h1>Distributed caching</h1> WOLFRAM also provides a distributed cache mechanism. There are several caches in the system. As well as one cache created implicitly per node, which is stored only on that node, and a global |
︙ | ︙ |