Obtaining the Code

NSU3D and the accompanying pre and post-processing tools are distributed as source code due to the wide variety of target operating systems and hardware that exist for High Performance Computing. There are two ways in which the code is given to end-users, by SVN export or by '.tar.gz' archived directory trees. If you have received the .tar.gz archives, then you should unpack them in the place where you would like them to reside. Assuming the .tar.gz files are in a Downloads folder in your home directory, then for a typical Linux/Unix system the process would go as follows, replacing /opt with your install directory of choice:

If your system tar command does not support inline gzip compression, then you will need to run gzip -d on each archive and then use tar xvf to unpack it. Alternately, if you have received valid credentials for accessing the NSU3D SVN repositories and you have a recent SVN client installed, you can get the codes via SVN checkout using the version tag that was provided with the repository access credentials. Once you have created an installation directory as above, the SVN checkout is as follows:

At this point, you should have the source code installed in these directories:

Building the Flow Solver

Prerequisites: The following packages should already be installed in a typical Linux/UNIX OS:

• sh/bash
• gnu make
• cpp
• sed
• awk

The following must be installed before building the software:

• a compatible Fortran90/95 compiler

If the solver is to be build for parallel execution then:

• an MPICH1 compatible MPI library and Fortran compiler wrappers
• the Fortran compiler must support OpenMP if OpenMP is desired for shared memory architectures

The NSU3D flow solver can be compiled with a number of optional parameters, as well as benefiting from architecture specific compiler optimizations. The code can also be built with OpenMP and/or MPI support for use on both shared and distributed memory supercomputers.

The included INSTALL_NSU3D.sh script will suggest default options for some compilers, but it may be necessary to use different options for certain hardware configurations. Note that not all compilers are guaranteed to build the code successfully, although the serial version should build with many recent Fortran90/95 compliant compilers.

If the source code has been installed as above, run the install script to generate the executable for your platform.

The script will then ask a number of questions to customize the configuration to your platform:

The script will then build the solver with the specified options and place the final executable in the EXECS subdirectory.

If the solver fails to build then it may be necessary to modify the compiler options or install any missing dependencies. To aid in investigating the failure, there will be a file called 'build.log' in the SOURCE directory. This file contains the compiler output and any errors that occured during the build. Please send this file along with a description of the host system, compiler, etc. to the developers when requesting assistance.

Enabling MPI Support

To build an MPI based parallel executable version of nsu3d, the MPI variable in the make file should be set to:

This has the effect of including all calls to the MPI message-passing library. In addition to this, the appropriate MPI library must be linked at load time. This can be usually be accomplished in two manners:

• A compile flag which specifies the linkage to the MPI library (usually -lmpi) can be added to the FFLAGS variables.
• A wrapper script which automatically sets environment variables and links to the appropriate library.

Such wrapper scripts are widely available on many systems. Under this approach, the variable FC which refers to the name of the FORTRAN compiler is changed to the name of the subject script. For example, the PGF compiler provides the script FC = mpif90 as a replacement to the regular compiler name FC = pgf90. Setting the FC variable in this manner is all that is required to compile and link an MPI-capable executable.

In order to run an MPI executable, the appropriate mpirun or mpiexec utility must be used. Thus, for example a 32 processor run is invoked as:

The -np parameter denotes the requested number of MPI processes. This is usually equal to the number of available processors, but may be a different number in principle. If -np is larger than the number of processors, multiple MPI processes will be assigned to each processor. If -np is smaller than the number of processors, idle processors will result.

In all cases, the -np parameter should be no larger than the number of partitions in the partitioned grid file, or the run will fail. On the other hand, the -np parameter can be smaller than the number of grid partitions. In this case, multiple grid partitions are processed sequentially by each MPI process. Efficient execution is achieved when the number of grid partitions is a multiple of the number of MPI processes. For example, a 32 way grid partition can be run efficiently on 1, 2, 4, 16, or 32 MPI processes.

Enabling OpenMP Support

For shared memory architectures where the MPI message-passing library is not available, an OpenMP executable can be employed.

In order to compile for OpenMP, the MPI variable should be set to:

in the make file. This has the effect of removing all MPI library calls in the source code. The appropriate compilation flags for OpenMP parallelization are then required in the FFLAGS variable for the fortran compiler. Usually, this is achieved using the -mp compilation flag, along with any other desired optimization flags.

should be used in all cases.

Once the code has been compiled, the number of processes can be specified at run-time using the NTHREAD variable in the input file. NTHREAD denotes the number of OpenMP threads to be used during parallel execution. On some systems it may also be necessary to set the OMP_NUM_THREADS environment variable to enable the requested number of threads to be employed.

If for such a reason NTHREAD threads cannot be spawned, nsu3d will terminate with a message to that effect. The number of threads should not exceed the number of partitions of the partitioned grid file. Although any number of threads can be specified at runtime, a larger number of threads will result in idle threads. A smaller number of threads will results in more than one partition being processed on each cpu.

For efficient execution, the number of partitions should thus be a multiple of the number of threads/processors employed. Thus a 32 way grid partition can be run efficiently on 1, 2, 4, 16, or 32 threads/processors.

OpenMP executables are run in the same manner as any other executables, by simply invoking the executable name along with the name of the input file:

Enabling Hybrid MPI-OpenMP Support

NOTE: Most users will not need to go through this step as Hybrid MPI-OpenMP support will be enabled when using the packaged installation script as described on this page: Building the Flow Solver

In order to build a hybrid MPI-OpenMP executable version of nsu3d, the procedure described for building an MPI based executable should be followed.

In addition to this, the appropriate compilation flags for OpenMP parallelization are required in the FFLAGS variable for the fortran compiler. Usually, this is achieved using the -mp compilation flag, along with any other desired optimization flags.

In order to run the hybrid MPI-OpenMP executable, the mpirun facility must be invoked as described previously:

Where the -np parameter denotes the requested number of MPI processes. In addition, the NTHREAD parameter in the input file must be set to the number of desired threads to be spawned under each MPI process. For example, NTHREAD = 4 in this case will result in a 8x4 = 32 way parallel job, using 8 MPI processes running 4 threads apiece.

The rules for number of threads and MPI processes in relation to the number of grid partitions remains similar to those described previously

Other Compile Time Options

NOTE: Most users will not need to go through these steps as these settings will be configured when using the packaged installation script as described on this page: Building the Flow Solver

The ARCH compile variable in the make file is used to specify conditional compilation for various machine architectures.

In general, ARCH can be set to several machine types (i.e. INTEL, SGI, IBM, T3E, SV1 etc..) with the default being ARCH = OTHER, which is used for any types other than those listed explicitly.

On CRAY machines, the appropriate ARCH setting also results in the input/output routines reading and writing IEEE standard unformatted fortran files which are compatible with other UNIX workstations and PCs.

The VEC variable enables (VEC=VEC_ON) or disables (VEC=VEC_OFF) vectorization. Performance will suffer if the appropriate setting is not used for the target machine architecture.

For most machines, VEC=VEC_OFF is employed, except for CRAY vector machines which requires VEC=VEC_ON

Two discretizations, one employing scalar artificial dissipation, the other matrix artificial dissipation (i.e. an upwind scheme) can be selected at compilation time. This is achieved through the use of the dflux_loop.h include file.

Scalar artificial dissipation is chosen by copying the dflux_scalar_loop.h file to dflux_loop.h and recompiling the entire source code.

Matrix artificial dissipation is chosen by copying the dflux_matrix_loop.h file to dflux_loop.h and recompiling the entire source code.

Matrix dissipation, which is equivalent to an upwind scheme, requires slightly more cpu time per iteration, but produces more accurate results on an equivalent grid. This is the preferred option. In cases where robustness is problematic, reverting to the scalar dissipation case may prove beneficial.

The values of the dissipation coefficients (VIS1, VIS2) should remain unaffected by the above choice of discretizations.

Building the Utility Suite

The prerequisites for building the NSU3D tools are the same as for the flow solver accept for the following:

• If you have the Tecplot I/O library then it can be used to produce binary Tecplot output from NSU3D results. The library should be placed in the SOURCE/libtecio sub-directory in the install path.
• A compatible C compiler is needed to build the metis partitioning library
• MPI support is not needed
• OpenMP support is not needed

If you do not have the Tecplot library then just leave the default TECIO_OFF

At this point the script will attempt to build each of the tools and place them in the EXECS directory. When the script completes, the following should be present:

• amg3d
• distf3d
• dmin3d
• kmetis
• makepoin1
• merge1
• merge2
• post_nsu3d
• pre_nsu3d
• PRENSU3D.sh
• setwallbc
• sliceslice tecform
• ugcombine
• ugform

If any of the above are missing, then it may be necessary to modify the compiler options or install any missing dependencies. To aid in investigating the failure, there will be a file called 'build.log' in the SOURCE directory for each of the tools. This file contains the compiler output and any errors that occured during the build. Please send this file along with a description of the host system, compiler, etc. to the developers when requesting assistance.

Configuring the Host System

Once the NSU3D solver and associated tools are compiled, the user should add the executables to their system PATH. This can be done either on a system or per user basis. As an example, the login script for the bash shell, .bashrc (in the root of each user's home directory) can be used to add the path for a specific user by adding the following lines: ​

The following command can be used to see if the path is set correctly:

It should return the location of the pre-processing script. If not, then it may be necessary to re-load the bash configuration:

If you are not using the bash shell or would like to set the system-wide path, then there are numerous on-line tutorials on the subject for Unix/Linux sytems.