1
The following instructions describe how to setup Myrinet and build
2
ARMCI to run over Myrinet using the Myricom GM interface and MPICH/GM.
5
- any other message-passing library is not supported
6
- This code was tested and is supported only on Linux Intel & Sparc
7
and Solaris/sparc platforms. If you have other configuration
8
please try if it works, otherwise please send a message to
9
<parsoft-support@emsl.pnl.gov>
11
1. Installation of Myrinet and the GM library (must have root privilege)
13
* To build gm on linux smp nodes, run configure with the following
14
options. For details, refer to README-linux.
16
--enable-linux-modversions
27
* Check the configuration
30
It lists the driver information and routing table.
32
2. Setting up the environment
34
* Note: We strongly recommend using GM version 1.2 or higher
35
* ARMCI on top of GM requires the following environment variables to
36
be set (Examples are taken on colony cluster of PNNL, using csh)
39
setenv GM_HOME /home/myrinet/gm-1.2/gm
40
setenv GM_INCLUDE $GM_HOME/include
41
setenv GM_LIB $GM_HOME/lib
42
setenv ARMCI_NETWORK GM
44
* For MPICH-GM the following additional environment variables need to be set
45
(again using examples of colony cluster of PNNL and csh)
47
setenv MPI_LOC /home/myrinet/mpich-1.2/mpich-gm
48
setenv MPI_LIB $MPI_LOC/build/LINUX/ch_gm/lib
49
setenv MPI_INCLUDE $MPI_LOC/include
50
set path = ($MPI_LOC/bin $MPI_LOC/build/LINUX/ch_gm/bin $path)
53
However, if you prefer to use MPICH compiler wrappers mpicc, mpif77 etc
54
these definitions are not needed. Also, since the default name for
55
the MPI library assumed by armci is -lmpi, you would need to
56
set LIBMPI="" or "-lmpich".
60
set a $HOME/.gmpi/conf file like this:
61
# .gmpi/conf file begin
62
# first the number of nodes in the file
64
# the list of (node,port) that make the MPI World
65
nb26.colony.emsl.pnnl.gov 2
66
nb26.colony.emsl.pnnl.gov 4
67
nb27.colony.emsl.pnnl.gov 2
68
nb27.colony.emsl.pnnl.gov 4
69
nb28.colony.emsl.pnnl.gov 2
70
nb28.colony.emsl.pnnl.gov 4
71
nb29.colony.emsl.pnnl.gov 2
72
nb29.colony.emsl.pnnl.gov 4
77
node files are created dynamically, but should have the same
78
format as the default .gmpi/conf
80
IT IS REQUIRED THAT MPI TASKS RUNNING ON THE SAME SMP NODE
81
BE NUMBERED CONSECUTIVELY
85
The current ARMCI implementation uses two GM ports.
86
If STATIC_PORTS is defined, they are called
87
ARMCI_GM_SERVER_RCV_PORT
88
ARMCI_GM_SERVER_SND_PORT
89
The default port numbers are 5 and 6, set in the file armci/src/myrinet.h.
90
Users can choose different ports by editing the definition, depending on
91
the system settings and at least not conflicting with the ports used by MPI.
93
If STATIC_PORTS is NOT defined, ARMCI will search for first two available
94
ports on up to two myrinet boards.
98
Go to armci/src and make test.x
100
Using mpich-gm: run it as regular mpi programs (make sure the mpirun
101
is come from mpich-gm. many systems have multiple mpi installed)
103
mpirun -np #procs test.x
105
This will read the default .gmpi/conf
107
4. Contact info and bug report
108
email: parsoft-support@emsl.pnl.gov
112
GM does not allow fork. This system call is normally used on Unix
113
in ARMCI to test how big a shared memory segment can be allocated.
114
This test is diabled under Myrinet/GM. It means that ARMCI can
115
use only shared memory segments predefined _SHMMAX in shmem.c
116
for a particular platform. If you increase that limit in the system,
117
_SHMMAX must be accordingly modiefied and ARMCI recompiled.
119
The current port or ARMCI to GM is not fully optimized yet.