Gaussian 16 for Intel Compiler with Intel MKL

Gaussian 在2017年公佈新版的Gaussian 16，並同時支援GPU的計算，雖然對於GPU計算一定要用PGI Compiler的部份頗有微詞，不過用GPU的效率實在不是很高，所以就算了吧。

跟Gaussian 09一樣，Gaussian 16在PGI Compiler下編譯完全無難度，但在Intel Compiler下編譯，就需要一些小功夫了。
為了以後改版要編又忘掉，寫個備忘吧！

重點有2個：

1. 編寫Makefile
    在x86_64的系統中安裝，要改的Makefile是i386.make這個檔案，以下是我編譯時的檔案，
     其中紅色的部份是重點，藍色是必須註解。

#
# Makefile for Gaussian 16.
#
#     Copyright (c) 1988,1990,1992,1993,1995,1998,2003,2009,2016,
#                Gaussian, Inc.  All Rights Reserved.
#
#     This is part of the Gaussian(R) 16 program.  It is based on
#     the Gaussian(R) 09 system (copyright 2009, Gaussian, Inc.),
#     the Gaussian(R) 03 system (copyright 2003, Gaussian, Inc.),
#     the Gaussian(R) 98 system (copyright 1998 Gaussian, Inc.),
#     the Gaussian(R) 94 system (copyright 1995 Gaussian, Inc.),
#     the Gaussian 92(TM) system (copyright 1992 Gaussian, Inc.),
#     the Gaussian 90(TM) system (copyright 1990 Gaussian, Inc.),
#     the Gaussian 88(TM) system (copyright 1988 Gaussian, Inc.),
#     the Gaussian 86(TM) system (copyright 1986 Carnegie Mellon
#     University), and the Gaussian 82(TM) system (copyright 1983
#     Carnegie Mellon University). Gaussian is a federally registered
#     trademark of Gaussian, Inc.
#
#     This software contains proprietary and confidential information,
#     including trade secrets, belonging to Gaussian, Inc.
#
#     This software is provided under written license and may be
#     used, copied, transmitted, or stored only in accord with that
#     written license.
#
#     The following legend is applicable only to US Government contracts
#     under DFARS:
#
#                        RESTRICTED RIGHTS LEGEND
#
#     Use, duplication or disclosure by the US Government is subject to
#     restrictions as set forth in subparagraph (c)(1)(ii) of the Rights
#     in Technical Data and Computer Software clause at DFARS
#     252.227-7013.
#
#     Gaussian, Inc., 340 Quinnipiac St., Bldg. 40, Wallingford CT 06492
#
#     The following legend is applicable only to US Government contracts
#     under FAR:
#
#                        RESTRICTED RIGHTS LEGEND
#
#     Use, reproduction and disclosure by the US Government is subject
#     to restrictions as set forth in subparagraph (c) of the Commercial
#     Computer Software - Restricted Rights clause at FAR 52.227-19.
#
#     Gaussian, Inc., 340 Quinnipiac St., Bldg. 40, Wallingford CT 06492
#
# Where to find this file when making executables:
#
BSDDIR = bsd
MAKELOC = -f $(BSDDIR)/g16.make
BSDDIR1 = ../bsd
MAKELOC1 = -f $(BSDDIR1)/g16.make
UTILDIR = ..
HLIBDIR = $(g16root)/hermes/lib
HSRC = $(g16root)/hermes/gxinterface
HINC = $(g16root)/hermes/include
HLIBS = $(HLIBDIR)/libdbapi.a $(HLIBDIR)/libsupp.a $(HLIBDIR)/libisam.a \
$(HLIBDIR)/libcbt.a $(HLIBDIR)/libutils.a $(HLIBDIR)/libgxchm.a
HFLAGS = $(CFLAGS) -I$(HINC) -D_POSIX_SOURCE
#
# The utility library:
#
#BLAS = blas-opt.a blas-f2c.a
#BLASL = -Wl,blas-opt.a -Wl,blas-f2c.a

BLASL = -Wl,$(BSDDIR)/libf77blas-em64t.a -Wl,$(BSDDIR)/libatlas-em64t.a
BLAS1 = $(BSDDIR)/libf77blas-em64t.a
BLAS2 = $(BSDDIR)/libatlas-em64t.a
BLAS = $(MKLLIBS) $(BLAS1) $(BLAS2)
GAULIBA = util.a
GAULIBU = util.so
GAULIB = $(GAULIBU) $(BLAS)
LINDALIBS = $(GAULIB) $(BLAS)
#
# Directory pointers only used for linking the profiling version:
#
GSDIR = .
GDIR = $(g16root)/g16
#
# Standard dimensioning definitions.
PCMDIM = -DDEFMXTS=2500 -DDEFMXBOND=12 -DDEFMXSPH=250 -DDEFMXINV=2500  -DDEFMXSLPAR=300 -DDEFMXSATYP=4
#CSIZE = 524288
#CSIZEW = 64
INCDIR =
INCDIRG = -I$(g16root)/g16
ACCFLAG =
PARMETH = -D_OPENMP_ -D_OPENMP_MM_
#FPARFLAG = -mp=nonuma
FPARFLAG =
PARFLAG = -DGAUSS_PAR -DGAUSS_THPAR $(PARMETH)
BLASFLAG = -DCA1_DGEMM -DCA2_DGEMM -DCAB_DGEMM -DLV_DSP
DEBUGP = -DCHECK_ARG_OVERLAP
I8CPP = $(I8CPP1) $(I8CPP2) $(I8CPP3) $(I8CPP4)
GAUDIM = 2500
GAUDIMA = $(GAUDIM)00
GAUDIMR = $(GAUDIM)0
GAUDIMS = $(GAUDIM)
CTDEBUG = -DDEFICTDBG=0
PROCTYPE =
NISEC = -DDEFISEC=16
NJSEC = -DDEFJSEC=128
NKSEC = -DDEFKSEC=128
NREPFD = -DDEFNREPFD=32
X86TYPE =
#USE_MPFUN90 = -DGAUSS_MPFUN90
DIMENSX = $(INCDIR) $(INCDIRG) -DDEFMAXRES=$(GAUDIMR) -DDEFMAXSEC=$(GAUDIMS) $(I8CPP) $(PARFLAG) $(DEBUGP) -DDEFMAXSHL=$(GAUDIMA) -DDEFMAXATM=$(GAUDIMA) $(PROCTYPE) -DNO_SBRK $(X86TYPE) \
  -DDEFMAXNZ=$(GAUDIMA) $(NREPFD) -DDEFNVDIM=257 -DR4ETIME \
  -DDEFARCREC=1024 -DMERGE_LOOPS -D_I386_ -DLITTLE_END -DUSING_F2C -DSTUPID_ATLAS \
  -DDEFMAXXCVAR=150 -DDEFMAXIOP=200 -DDEFMAXCOORDINFO=32 -DDEFMAXSUB=80 -DDEFMAXCHR=1024 -DDEFMOMEGA=5 -DDEFNOMEGA=6 -DDEFMAXXCNAME=25 -DDEFLMAX=13 -DDEFMINB1P=100000000 -DDEFXGN3MIN=1 $(NISEC) $(NJSEC) $(NKSEC) -DDEFN3MIN=10 -DDEFNBOMAXBAS=10000 -DDEFMAXHEV=2000 -DDEFCACHE=128 \
  -DDEFMAXLECP=10 -DDEFMAXFUNIT=5 -DDEFMAXFFILE=10000 -DDEFMAXFPS=1300 -DDEFMAXINFO=200 \
  -DDEFMAXOP=384 -DDEFMAXTIT=100 -DDEFMAXRTE=4000 -DDEFMAXREDTYPE=3 -DDEFMAXREDINDEX=4 -DDEFMAXOV=500 -DDEFMXDNXC=8 -DDEFMXTYXC=10 $(CTDEBUG) -D_ALIGN_CORE_ \
  $(BLASFLAG) -DO_BKSPEF -DSETCDMP_OK $(PCMDIM) -DGCONJG=DCONJG -DGCMPLX=DCmplx -DGREAL=DREAL -DGIMAG=DIMAG -DEXT_LSEEK -DAPPEND_ACC $(ACCFLAG) $(USE_MPFUN90)
#
# These commands are converted to "on machine command" for remote-
# control compilation.
#
RUNF2C = f2c -kr -T. -R -Nx400 -Nn1604 -NL800
RUNCC = icc -openmp -axCORE-AVX2 -mkl
RUNAR  = ar
RUNRAN = gau-ranlib
RUNCPP = gau-cpp
RUNFSP = gau-fsplit
RUNMAKE = make
#TIME = -Mreentrant -Mrecursive -Mnosave -Minfo -Mneginfo -time
#NOMCOPY = -Hx,8,0x8000000
#VECTOR4 = ,prefetch,sse -fastsse -Mscalarsse
#VECTOR = -Mvect=assoc,recog,noaltcode,cachesize:$(CSIZE)$(VECTOR4)
#MACHTY = p7-32
#MACH = -tp $(MACHTY) $(TIME)
#OPTOI = -m32 -march=i486 -malign-double 
#GCCOPTS = -ffast-math -funroll-loops -fexpensive-optimizations
OPTFLAGO = -O2 -unroll
# Flags for portland compiler.
#
I8FLAG = -i8
R8FLAG = -r8
MMODEL = -mcmodel=medium
#PGISTATIC = -Bstatic_pgi
#PGNAME = pgf77
PGNAME = ifort -openmp
SPECFLAG = -auto -axCORE-AVX2 -shared-intel -static-libgcc -no-prec-div -ftz -pad -fpp -mkl
GPUFLAG = $(GPUFLAG1) $(GPUFLAG2) $(GPUFLAG3)
RUNF77 = $(PGNAME) $(PGISTATIC) $(I8FLAG) $(R8FLAG) $(MMODEL) $(DEBUGF) $(SPECFLAG) $(GPUFLAG)
F2CLIB =
GPULIB2 =
#SYSLIBS = -lpthread -lm -lc $(GPULIB2)
GPULIB1 =
GPULIB2 =
NUMALIB = -L/opt/inte/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl
LIBS = $(GPULIB1) $(GPULIB2) $(NUMALIB) $(SYSLIBS)
UNROLL = -unroll
TWOH =
PC64 = -pc64
DIMENS = $(DIMENSX) $(TWOH)
FNOOPT = $(FPARFLAG) $(PROFFLAG) -O0 $(MACH) -g
FNOOPT64 = $(FNOOPT) $(PC64)
FOPT1 = $(FPARFLAG) $(PROFFLAG) -O1 $(MACH)
FOPT2 = $(FPARFLAG) $(PROFFLAG) -O2 $(MACH)
FOPT2UN = $(FOPT2) $(UNROLL)
FOPT2VC = $(FOPT2) $(VECTOR)
OPTFLAG = -O3 $(UNROLL) $(VECTOR)
LINK1 =
LINK2 =
EXTCFLAGS =
FFLAGS = $(FPARFLAG) $(PROFFLAG) $(MACH) $(OPTFLAG) $(LINK1) $(LINK2)
CFLAGS = $(DIMENS) $(OPTFLAGO) $(PROFFLAG) $(GPUFLAG) $(EXTCFLAGS)
LFLAGS = $(FFLAGS)
EXTOBJ1 =
EXTOBJ2 =
EXTOBJ = $(EXTOBJ1) $(EXTOBJ2)
TESTRTO =

---以下都不會變動，省略---

一樣要注意的是，CSIZE跟CSIZEW對編好的程式效能影響很大，這兩個參數指的是CPU Cache memory的大小，單位是kW，這邊先把它們註解，然後把DDEFCACHE設成128 [原本是DDEFCACHE=$(CSIZEW)]。

要達到最大效能，DDEFCACHE必須是CPU每一個核心所能用的cache size的一半。

比方說：

Intel Xeon E5-2650Lv3(12 cores/30MB)，(30*1024)/(8*12*2) = 160

Intel Xeon Scalable Gold 6130(16 cores/22MB Cache)，(22*1024)/(8*16*2) = 88

以上面這兩個例子，在E5-2650Lv3的機器上就要用"DDEFCACHE=160"，在Gold 6130的機器上就要用"DDEFCACHE=88"。

2. 修改mdutil.c

這個檔案位在bsd下面，必須作點小修正。

在g16.a01的版本中，要改成下面這樣:

#ifdef __x86_64

#define MAX_IO (2000*1024*1024)

#endif

#include <sys/times.h>

#define NEED_AND

#define NEED_ISHFT

#define NEED_GSR48

#define NEED_PUTENV  /* need for Intel Compiler */

#include <unistd.h>

#include <errno.h>

#include <time.h>

#endif

大概在242行，多加一個"#define NEED_PUTENV"。

把Makefile跟mdutil.c都弄好後，就是編譯啦。

另外，由於Gaussian的原始碼有部份相當的舊，並沒有全面改寫為新版的Fortran規範。

所以當使用Intel Compiler 2017 update4或以上版本時，會因為Intel Compiler遵循嚴謹的Fortran 2008規範，而導致在編譯過程會出現以下錯誤。

Multiple objects from the same EQUIVALENCE set may not appear in a COMMON block

這個狀況會發生在編譯l702.exe跟newzmat這個Utility時。

除非系統環境沒有辦法，不然真的沒必要用太新的編譯器去編譯Gaussian，對於效能並沒有幫助。

但真的只能用最新的編譯器時，解決方法是稍微改一下Source Code：

l702.F : 

在3911行，把EQUIVALENCE (FXI,HES(1))註解掉，也就是多加一個英文字"C"改成下面這樣

C      EQUIVALENCE (FXI,HES(1))

這邊一定要注意，英文字C一定要在一整行的第一個字元

newzmat.F :

在第339行，把

Equivalence (NZ,NZNVar(1)), (NVarZM,NZNVar(2)), (ToAng,PhyCon(1))

改成

Equivalence (NZ,NZNVar(1)), (ToAng,PhyCon(1))

大致上就是醬啦。