Skip to content

Commit

Permalink
use only cmq for hpc
Browse files Browse the repository at this point in the history
  • Loading branch information
mschubert committed Jan 9, 2017
1 parent a741a3f commit 87f5c1d
Show file tree
Hide file tree
Showing 6 changed files with 2 additions and 441 deletions.
39 changes: 1 addition & 38 deletions hpc/README.md
Original file line number Diff line number Diff line change
@@ -1,41 +1,4 @@
High performance computing / LSF jobs
=====================================

Perform function calls in LSF jobs. The module will start `n_jobs` LSF jobs (workers) with the
memory requirements specified, and then send `<i>` function calls to those workers.

This is done entirely on the network and without temporary files (unless `log_worker=TRUE`),
so there is no strain on the file system apart from starting up R once per LSF job.

The module also performs load-balancing, i.e. workers that get their jobs done faster will also
receive more function calls to work on. This is especially useful if not all calls
return after the same time, or one worker has a high load. For long running jobs use `n_jobs=<i>`.

It is based upon the [rzmq package](https://github.com/armstrtw/rzmq) and the
[ZeroMQ library](http://zeromq.org/) that is also used for workers in IPython.

It also uses the [ulimit package](https://github.com/krlmlr/ulimit).

The function supplied **must be self-sufficient**, i.e. load libraries and scripts.

### Custom setup

Currently, only LSF is supported as a scheduler. Adding others should be simple, but will
only be implemented if there is a need for it.

If not at EBI, you may need to adjust the `LSF.tmpl` file according to your needs,
especially for the queue and custom resources.

### `Q()`

```r
hpc = import('hpc')
s = function(x) x
hpc$Q(s, x=c(1:3), n_jobs=1) # list(1,2,3)
```

```r
t = function(x) sum(x)
a = matrix(3:6, nrow=2)
hpc$Q(t, a, n_jobs=1) # splits a by columns: list(7, 11)
```
This module now uses the [`clustermq`](https://github.com/mschubert/clustermq) package.
2 changes: 1 addition & 1 deletion hpc/__init__.r
Original file line number Diff line number Diff line change
@@ -1 +1 @@
export_submodule('./master')
Q = clustermq::Q
163 changes: 0 additions & 163 deletions hpc/master.r

This file was deleted.

61 changes: 0 additions & 61 deletions hpc/process_args.r

This file was deleted.

124 changes: 0 additions & 124 deletions hpc/template_LSF.r

This file was deleted.

Loading

0 comments on commit 87f5c1d

Please sign in to comment.