2
This is a preview of the Makeflow workflow engine.
3
It is still under development and not fully documented yet.
5
Makeflow uses a simplified version of Makefiles to represent
6
workflows for distributed computing. It is not tied to any
7
particular batch system, and it currently works with Unix
8
processes, Condor, SGE, and a custom system called Work Queue (wq).
10
The file example.makeflow in this directory contains an example
11
workflow that takes an image, swirls it around in 4 permutations,
12
and joins the results into a gif animation.
14
Looking at the source, you should see that it looks almost identical
15
to a Makefile. The main difference is that Makeflow is STRICT:
16
You must declare all source files needed by a rule, and all of the
17
target files created by a rule, because Makeflow will use that
18
information to create batch jobs. You cannot have a phony rule
19
like "clean:" because such a rule does not actually create a
22
To run it on your local machine as if it were a Makefile:
23
makeflow example.makeflow
25
To clean up the results of the run:
26
makeflow -c example.makeflow
28
To cause Makeflow to submit jobs to Condor:
29
makeflow -T condor example.makeflow
31
To submit Makeflow as a Condor job that submits more Condor jobs:
32
condor_submit_makeflow example.makeflow
34
(In this configuration, makeflow behaves like dagman: it can
35
recover from a failure of the submissino machine. We haven't
36
implemented a clean condor_rm of a makeflow yet, but it is quite doable.)
38
You will notice that a workflow can run very slowly if you submit
39
each batch job to a system like Condor, because it typically
40
takes 30 seconds or so to start each batch job running. To get
41
around this limitation, we provide the Work Queue system, which
44
To start your Makeflow with Work Queue on port 9123:
45
makeflow -T wq -p 9123 example.makeflow
47
Then, start as many workers as you like on whatever machines you
48
have available. You can ssh to another machine and run:
49
work_queue_worker servername 9123
51
Or, you can submit 100 workers to Condor like this:
52
condor_submit_workers servername 9123 100
54
For more information, please check out our manuals and mailing list here:
55
http://www.cse.nd.edu/~ccl/software/help.shtml
added
removed
makeflow/example/example.README
