Submit a Fluent job on ADA

There are majorly 3 ways with which Ansys Fluent can be used on ADA.

1. Using a VNC desktop sesison via Open OnDemand.

2. Using an interactive session.

3. Using a batch mode submission.

Note that the former two will allow for a GUI based interaction, while the last option is a no-GUI option (a.k.a batch mode) job.

When there is only a few models, required to be run on ADA, with not much changing in between those models, a GUI based option can be explored with ADA.

However, if there are several configurations of the model that is to be simulated, (perhaps of the same model or otherwise), a batch mode submission (without GUI) should be used.

Note that this documentation doesn’t factor in the workbench based workflows that import Fluent job from various other standalones present in Ansys Workbench. This is simply becuase Workbench based Ansys usage is currently not supported for ADA yet. Do write to us if you would need help setting up the same for a more bespoke solution.

A simple workaround suggested for general usage for the same is that the CAD or Mesh of the model or assembly should be exported separately as a file (Fluent readable format). This will allow Ansys Fluent to import / read this file separately and can be easily done via both, batch mode for no-GUI job submissions, as well as GUI based jobs.

GUI based jobs

VNC session and Visualization node will be the two preferred method to use a GUI for Ansys Fluent.

Using a VNC desktop session via Open OnDemand

This will be a GUI based usage. Which means instead of your local system/laptop, you will be using Fluent on ADA to perform simulations.

1. Head to the Open OnDemand website.

2. Once the page opens, go to Interactive sessions -> VNC desktop sessions

3. One can now book the appropriate resource for the Ansys Fluent based on following

   1. Set the number of hours for which you want to book the resource. (Note that once this time is over, the VNC session as well as everything that was being used here will be closed. Save your work accordingly or book the time accordingly.)

   2. Set the number of cores for your job. (Note that this is on defq or hmemq partitions, which mean each node has a maximum of 96 cores. However, more number of cores can be booked, 96 is the number of CPUs for 1 node. And going to two nodes will increase the time for computation as now Fluent has to manage/copy files between two or more nodes. See more on scaling here Calculate number of cores required for Fluent job.

   3. Set the amount of memory required for your job. (The thumb rule for setting the memory atleast for the defq is 4 times the number of cores. For hmemq this can be higher.)

   4. Set the partition (either defq or hmemq).

4. Launch the VNC desktop session by clicking “Launch”.

5. Once the resource becomes available, click “Launch VNC session”, and you will be taken to a linux based desktop like session, along with an email that your session has started.

6. Right click anywhere on the desktop screen and click “Open terminal here”. This will open up a shell with your login node visible.

7. Change to the directory which contains the Ansys Fluent files.

8. Load the fluent module with ,

module load ansys-uoneasy/2022R2

9. Start Fluent from the terminal.

fluent

10. The following screen will start indicating that Fluent can now be used as a regular software. Note that in this screen, the number of processors need to be equal to the number of cores booked in the VNC session booking from step 3.b. above.

11. Enjoy using Ansys Fluent on ADA.

Using an interactive session

This session implies that you have booked the requested resource exclusively for yourself and can use the resource as a desktop equivalent of ADA.

To do this,

1. Go to the directory in your home directory on ADA which contains the Fluent files.

2. Load the Ansys module,

module load ansys-uoneasy/2022R2

3. Book an interactive session. Using the following command accordingly,

srun --partition=defq --nodes=1 --ntasks-per-node=48 --mem=192g --time=2:00:00 --pty /bin/bash

# change partition flag to the appropriate partition accordingly.
# Thumb rule for nodes is kept as 1, as going to 2 nodes includes a communication overhead, slowing down computations.
# Thumb rule for mem flag is 4 times the number of cores on that partition for 1 node. Say for example, for defq partition 1 node has 48 cores, so 48 x 4 = 192G. Same applies for other partitions as well, however are slightly different for hmemq partition and the GPU partitions.
# Recommended time is also for 2 hours, however, remember that after this scheduled time, the allocated resource will be closed, which means, any notebook running during that time will also close and will be difficult to recover. Set the time accordingly.
# Using a GPU partition for Ansys fluent is discouraged, as there is little documentation for the same, including license limitations.

This means that 1 full node with 192G memory (48 cores with 4GB each) is requested for 2 hours time on the defq partition. Once the resource becomes available and you are inside the interactive session, the shell will change to the name of the corresponding compute node, i.e.

$ userid@ hpclogin02 (Ada)

## will change to the booked resources, say copmp004 becomes available.

$ userid @ comp004 (Ada)

4. Type in the fluent command to open the Fluent session.

fluent

Note that as these partitions do not have hardware acceleration, there is a possibility that the session may observe some lag in between mouse clicks. To use hardware acceleration for Ansys Fluent, one could also explose remote visualization which has been created for post processing purposes, but can also be used for a GUI based job for Fluent. The steps to run job on remote visualization can be seen from here Via Visualization Nodes.

Non-GUI based jobs

As mentioend before, non-GUI based jobs mean that Fluent session is used and operated only via previously tested commands that set the case and run the simulation.

For this kind of Ansys Fluent usage, one would require three things.

1. A journal file.

2. A Fluent file (this could be a CAD file or a Fluent msh file).

3. A Slurm script that calls the fluent executable and sets the hardware resource for an HPC run.

Lets take a look at each of them one by one.

Journaling

We look at an example of the following case, where a simple case has been demonstrated using the journaling features within Fluent. This case is of a mixing elbow case.

The boundary conditions for this mixing elbow case have been demonstrated in the image below.

The problem to be considered is shown schematically. Cold fluid at 20° C flows into the pipe through a large inlet, and mixes with a warmer fluid at 40° C that enters through a smaller inlet located at the elbow. The pipe dimensions are in inches and the fluid properties and boundary conditions are given in SI units. The Reynolds number for the flow at the larger inlet is 50,800, so a turbulent flow model will be required. A 3D demonstration of this boundary condition and the case is shown from the image below.

This example will create settiings and submit this mixing elbow case and run the simulation on ADA.

A journal file as indicated below, essentially contains steps that one wants to be performed while setting up a Fluent case. This journal file should ideally contain all the global variable definitions, meshing conditions, surface-names etc. The level of detail for each of the tasks that should be scripted in journal depends on user’s tasks. These commands are usually created in Fluent’s console. See How to construct a journal file for Fluent batch jobs for how to use Fluent console to create a journal file for your Fluent jobs.

Attention

It is highly recommended to test the journal file in the Fluent console available in the GUI on a local machine before trying any tests on an HPC system. This step will help you solve any errors related to the journal file and can be corrected in GUI. Once the journal is without any errors, the same can be run on HPC

Corresponding Fluent’s console’s output has been indicated in the tab, indicating a successful opening of Fluent, setting up node_list that would run Fluent job, running for a specified number of iterations (150 in this case) and the model converging, finally reporting the system stats for the job.

Journal fileAnsys Console output

/file/read/mixing-elbow-mesh.msh.gz
/mesh/check
q
;; define/models/Enable energy model? / Compute viscous energy dissipation? / Include pressure work in energy equation? / Include kinetic energy in energy equation? / Include diffusion at inlets?

/define/models/energy yes no no no no
q
q

/define/models/viscous/ke-standard yes
q
q

/define/models/viscous/near-wall-treatment/enhanced-wall-treatment yes
q
q

/define/materials/copy-by-formula fluid h2o<l>

/define/materials/change-create water-liquid water-liquid yes constant 1000 yes constant 4216 yes constant 0.677 yes constant 8e-4 no no no

/define/materials/list-materials
q

; assign water (created material) to the cell zone.
/define/boundary-conditions/fluid fluid yes water no no no no 0 no 0 no 0 no 0 no 0 no 1 no yes no no no

; set boundary conditions.
define/boundary-conditions/velocity-inlet inlet-large no no yes yes no 0.4 no 0 no 293.15 no no no yes 5 4

define/boundary-conditions/velocity-inlet inlet-small no no yes yes no 1.2 no 0 no 313.15 no no no yes 5 1


define/boundary-conditions/pressure-outlet outlet yes no 0 no 300 no yes no no no yes 5 3 yes no no no

define/boundary-conditions/wall wall 0 no 0 no no no 0 no no no no 1

;24=coupled, 20=simple, 21=simplec, 22=piso
/solve/set/p-v-coupling 20

/solve/set/warped-face-gradient-correction/enable no


; create surface definition.
/solve/report-definitions/add outlet-temp-avg surface-massavg surface-names outlet () average-over 1 field temperature report-type surface-massavg
q

/solve/convergence-conditions/conv-reports/add outlet-temp stop-cri 1e-5 initial-values-to-ignore 20 active no print yes report-defs "outlet-temp-avg"
q
q

/solve/convergence-conditions/conv-reports/add outlet-temp stop-cri 1e-5 initial-values-to-ignore 20 active yes print yes report-defs "outlet-temp-avg" q q q q

/solve/convergence-conditions/conv-reports/edit outlet-temp active yes
q
q


/solve/initialize/hyb-initialization
/iterate 150

/file/write-case-data ,
/display/save-picture residuals%i.jpg

/report/system/time-stats
/report/system/proc-stats

As our commands are set in place that can save an image, we see the image of residuals as shown below. The same can be done for any surface average property pre-defined with the journal commands.

Meshing

A mesh file is the file containting the mesh of the geometry you want to simulate in Fluent. One could very well use a CAD file (Linux Fluent readable format) itself such as an STL or a STEP file in place of a mesh file. In such a case, a journaling file will need to file had additional steps for meshing such as defining cell types, surface name tags, zone details, maximum and minimum cell size, boundary layer details etc. In case the starting point is a CAD file, the journal file will also update accordingly with the steps that my look as follows,

1. Import CAD file.

2. Set up boundary names and relevant settings.

3. Set up meshing parameters necessary for the model, such as min size, max size of cells, first boundary layer length etc.

4. Mesh

5. Switch to Fluent solver.

6. Set up CFD parameters such as models, materials, boundary conditions etc.

7. Run Fluent solver.

Steps 6 and 7 are indicated via an example in the journal file indicated above.

SLURM

Slurm job submission script (e.g. named as say, run_ansys.slurm) is the script that will be submitted to the scheduler to run your Fluent jobs. A sample script is shown:

Slurm

#!/bin/bash
# SLURM job script to run Ansys fluent in parallel on ADA
#SBATCH --job-name=myjob              ## names the case to view later.
#SBATCH --nodes=1                     ## requested number of nodes.
#SBATCH --ntasks=48                   ## requested number of cores.
#SBATCH --partition=hmemq             ## requested partition.
#SBATCH --time 0:10:00                ## requested time limit

module load ansys-uoneasy/2022R2 ## Load the correct version of Ansys (currently 2022R2)

echo "#######################################################################" >> out.txt
echo No of tasks: $SLURM_NTASKS >> out.txt
echo "#######################################################################" >> out.txt
echo

node_list=$(scontrol show hostname ${SLURM_NODELIST} | sort -u) #finds the details of the acquired node

echo $node_list >> out.txt

echo
echo "fluent commands start from here!!!" >> out.txt
echo

## The master fluent command that opens fluent in no-gui format on HPC system.
fluent 3ddp -ssh -t$SLURM_NTASKS -mpi=intel -pinfiniband -cnf=$node_list -gu -i setcase.jou >> out.txt

scontrol realease $SLURM_JOB_ID

The directive: #SBATCH --ntasks=40

represents the CPU cores you want to use for your simulations. Note that this is a function of the partition that will run your job. Some partitions on ADA have 96 CPU cores per node. In the above example, the defq partition, which is the default partition, has 96 cores, out of which 48 has been requested on 1 node. If more resources are requested then the avilable resource-based error will be prompted. See the following for meaning of various errors.

Reason	Description
Resources	All suitable resources are in use
Priority	There are higher priority jobs in queue
AssocGrpCPULimit	The maximum number of CPU cores allowed for running jobs has been reached
QOSMinMemory	The memory has not been defined, or has been set too low with ‘–mem’ option
QOSNotAllowed	An inappropriate partition has been selected
ReqNodeNotAvail	An inappropriate partition has been selected

The directive: #SBATCH --nodes=1

requests one node. For example, for the defq partition, which has 96 CPU cores per node:

SBATCH --ntasks = 200 ## Requesting 200 (>192) cores, with 2 nodes on defq will give resource error

SBATCH --nodes = 2 ## Requesting 1 node and 100 cores will also be  resource error, as defq has 96 cores per node.

Hence it is recommended to check the upper limit of the partition you are submitting to, and then select the node and ntasks for the job. The default partition will be defq which has 96 CPU cores per node.

The directive: #SBATCH --time =0:10:00

requests the time for which the job should run on the partition. In this example, it is set to 8 hours. Note that if your job exceeds this time, the job will be terminated by SLURM.

The meaning of the fluent’s executable call flags are as follows,

• 3ddp, uses 3-dimensional, double precision model of the fluent solver.

• -ssh is using the SSH protocol to launch MPI processes.

• -meshing: This flag opens the fluent meshing. If the user wants the solver, simply remove this flag. This is also done in this example, because the starting point is a mesh import to Fluent solver.

• -$SLURM_NTASKS requests the --ntasks value (an integer stored in the shell variable $SLURM_NTASKS) to the solver, so for the above example --ntask, this flag will become -t40.

• -mpi=intel This would be required if you have more than one node or -ntasks>1, and implies it will be using the Intel MPI implementation.

• -pinfiniband Required with -mpi=intel. Implies that Fluent will use the Infiniband interconnect.

• -cnf is the node list and can be used as it is.

• -gu This is the batch mode flag, which means a GUI will “not” open when you ask Fluent to open. Removing this flag will open Fluent GUI.

• -i Input a journal file with .jou extension.

• journal_run.jou The journal file that will perform the steps mentioned in it inside fluent.

• out.txt This outputs the print statements of the fluent console to a text file, which can be tracked in real-time with tail -f out.txt command once your submission has been made.