===================== OpenMP Threads = 12 ===================== Memory Usage: VmRSS = 123.566; VmSize = 5893.797; VmPeak = 5893.797 (MB) WriteFields activated ------------------------- WriteSurfFlag activated ------------------------- .probe file exists PW.regular file is missing =================================================================== INPUT DATA =================================================================== Penalty: 2.5000000000e-02 Basis Polynomial Order: 1 PlaneWave Excitation (0)TH (1)Gauss (2)Neuman: 1 Port Waveform (0)TH (1)Gauss: 0 Group File Name: PW Maximum Frequency (MHz): 3.0000000000e+03 Tdelay: 2.5000000000e-09 Tau: 5.0000000000e-10 Final Time: 8.0000000000e-09 Polynomial basis order: 2 Regular Mesh File: false WriteFields Mode: true WriteProbes Mode: true write_AnalyticalIncidentProbes Mode: false Sampling Rate: 1.2500000000e+01 GPU Mode (FinalTime): 8.0000000000e-09 =================================================================== MaxPoint = (0.0000000000e+00, 0.0000000000e+00, 0.0000000000e+00) MinPoint = (0.0000000000e+00, 0.0000000000e+00, 0.0000000000e+00) readNODE ==================================================================================================== PLANEWAVE BOUNDARY CONDITION ==================================================================================================== PlaneWaveType : 1.000000 0.000000 0.000000 0.000000 1.000000 0.000000 0.000000 0.000000 -0.200000 Unit : 1.000000 Name : Excitation magE : 1.0000000000e+00 Theta : 0.0000000000e+00 Phi : 0.0000000000e+00 POL : (0.0000000000e+00, 1.0000000000e+00, 0.0000000000e+00) r0 : (0.0000000000e+00, 0.0000000000e+00, -2.0000000000e-01) ==================================================================================================== readBC readTETRA totalObjNum: 1 Reading material properties from file: freespace readMaterial Memory Usage: VmRSS = 320.332; VmSize = 6090.340; VmPeak = 6090.340 (MB)  =========================== Make Arrays =========================== edgeCNT == 255014 makeEdgeArray totalFaceCount == 428632 makeFaceArray =========================== Memory Usage: VmRSS = 1073.941; VmSize = 6844.598; VmPeak = 6844.598 (MB) Initialize Octree that organize teterhedrals elements in space ========================================================== Compute AABB for tetrahedral Compute global bounding box Global Bounding Box: xmin = -0.200, xmax = 0.200 ymin = -0.200, ymax = 0.200 zmin = -0.200, zmax = 0.200 Max Range = 0.400 | Wavelength = 0.100 Compute octree with octree depth = 1 Octree build completed ========================================================== Memory Usage: VmRSS = 1086.434; VmSize = 6857.117; VmPeak = 6857.117 (MB) readPROBE Compute the Barycentric coordinates of the Probes readPROBE took 0.000 seconds. -------------------------------------- Write Surface Fields / Currents makeOutputSurfMesh -------------------- Reading Tri surface mesh ./PW_out.tri -------------------- Compute Normals -------------------- -------- Summary --------- Scale: 1.000 Number of Nodes: 2580 Number of Triangles: 5156 First Triangle: 0,1,2 Last Triangle: 2524,2561,2537 -------------------- Compute the Barycentric center of the nodes Completed -------------------- makeOutputSurfMesh took 0.605 seconds. -------------------------------------- ======================================== PlaneWave BC Detected ======================================== Generating InterSurf Mesh with 1000000 InterFaceNum == 5156 FaceNum == 5156 ->faceCNT == 5156 nodeNum == 2580 ->nodeCNT == 2580 makeInterSurfMesh pwFaceNum == 6048 planeWaveMesh->faceCNT == 6048 nodeNum == 3026 planeWaveMesh->nodeCNT == 3026 makePlaneWaveMesh ======================================== Memory Usage: VmRSS = 1086.434; VmSize = 6857.117; VmPeak = 6857.117 (MB) AssignExcitParamToFace AssignMaterialProperties ====================================================== Total number of TetraHedra ====================================================== Total number of TetraHedra := 211515 Total number of P2 TetraHedra := 211515 Total number of Interior TetraHedra := 205709 Total number of AbcCount TetraHedra := 5806 Total number of Port/PlaneWave TetraHedra := 5806 ====================================================== AssignTetraFlags TIMER:: DGTD: read geometry model took 2445 milliseconds ================= Dimensions ================= dimE = 6266388 dimH = 6345450 ================= DG_AssignOffsets SetUpMatrixFree ============================================== NUMBER OF DEGREES OF FREEDOM ============================================== Global Number of dof is 12611838 Global Matrix dim is (w/o compress) 12690900 ============================================== numberDofs ======================================================== LocalTimeSteppingClassPartioning ======================================================== Class Factor: (2m + 1), m = 1.00000000000000000000e+00 Calculating Time steps  Finished: 0.000000 %  Finished: 9.999764 %  Finished: 19.999527 %  Finished: 29.999291 %  Finished: 39.999054 %  Finished: 49.998818 %  Finished: 59.998582 %  Finished: 69.998345 %  Finished: 79.998109 %  Finished: 89.997872 %  Finished: 99.997636 % Get_dt_min = 3.61011869375268129183e-13 Get_dt_max = 2.16683133176126196762e-12 Starting class partitioning N_class: 2 Number of Tetra in class: 0 = 134129 Number of PML Tetra in class: 0 = 0 ------------------------------------------------------------- Number of Tetra in class: 1 = 77386 Number of PML Tetra in class: 1 = 0 ------------------------------------------------------------- Total Number of PML Tetras = 0 ----------------------- regularCNT = 1 regularCNT_Normal = 0 regularCNT_PML = 0 NumGroups = 5 Class 0 | PML index = 134129 Class 1 | PML index = 77386 ClassExcitationCount[0] = 3 ClassTetraOffset[0] = 0 ClassPMLTetraOffset[0] = 134129 ClassExcitationCount[1] = 5803 ClassTetraOffset[1] = 134129 ClassPMLTetraOffset[1] = 211515 excitationFaces = 6048 ======================================================== LocalTimeSteppingClassPartioning Memory Usage: VmRSS = 1147.074; VmSize = 7561.117; VmPeak = 7561.117 (MB) TIMER:: DGTD: prepare for computation took 277 milliseconds tetraCNT = 211515 TIMER:: CPU Matrices Evaluation took 28697994 microseconds GetMatrices PlaneWaveBCFlag = true exciCNT = 5806 nonregularCNT_Normal = 211515 nonregularCNT_PML = 0 num_elements_regular_PML = 148833816 -------------------------------------------------------------------------------------------------- regularCNT_Normal = 0 totalRegularNeighFaceCnt = 0 Complete regular matrices preparation -------------------------------------------------------------------------------------------------- regularCNT_PML = 0 totalRegularPMLNeighFaceCnt = 0 Complete regular PML matrices preparation -------------------------------------------------------------------------------------------------- Neighbor matrices preparation tetraCNT = 211515 cntAux = 211515 ============= PML ============= PML Complete Neighbor matrices preparation neighCNT = 834856 -------------------------------------------------------------------------------------------------- Excitation preparation exciCNT = 5806 ========== FILLING Irregular =============== Begin irregular CuBLAS preparation N_class = 2 irregularTetras = 211515 nonregularCNT_Normal = 211515 irregularTetras = 211515 exciCNT = 5806 -------------------------------------------------------------------------------------------------- nonregularCNT_PML = 0 -------------------------------------------------------------------------------------------------- ============================================================================================ Category Buffer Size [GB] -------------------------------------------------------------------------------------------- Excitation InvE 0.020902 Excitation InvH 0.020902 Excitation nd_coords_tet 0.000279 Excitation nd_coords_face 0.000218 Excitation mapE (int8) 0.000174 Excitation mapH (int8) 0.000174 Excitation ExcitationFacesNum (int) 0.000024 Excitation Z_face_pw 0.000024 Excitation ExcitationFacesCnt (int) 0.000023 Excitation ExcitationFacesOffset (int) 0.000023 Propagation Neigh1E (irreg) 1.202193 Propagation Neigh2E (irreg) 1.202193 Propagation Neigh1H (irreg) 1.202193 Propagation Neigh2H (irreg) 1.202193 Propagation Loc1E (irreg) 0.761454 Propagation Loc2E (irreg) 0.761454 Propagation Loc1H (irreg) 0.761454 Propagation Loc2H (irreg) 0.761454 Fields/State En 0.025382 Fields/State En1 0.025382 Fields/State Hn12 0.025382 Fields/State Hn32 0.025382 Neighbors auxFieldOutput 0.063763 Neighbors NeighMap (int) 0.040073 Neighbors auxFieldInput 0.025505 Neighbors Neighbours (int) 0.000846 Neighbors NeighboursOffset (int) 0.000846 -------------------------------------------------------------------------------------------- TOTALS Excitation 0.047017 TOTALS Propagation 8.640045 TOTALS Fields/State 0.111680 TOTALS Neighbors 0.144137 TOTAL (est.) 8.942879 -------------------------------------------------------------------------------------------- GPU Memory Free / Total [GB]: 12.09 / 12.61 ============================================================================================ Non regular PMLTetras_total = 0 GPU set up correctly ============================================= == Running CUDA Implementation (Non-Heavy) == ============================================= ========================================== PERFORMING INFORMATION ========================================== Final Time(sec) = 0.000000008000000 Time Step, dt(sec) = 0.000000000001083 Number of Tetrahedra = 211515 Number of Classes = 2 Number of Time Steps = 7387 LocTimeSteps[0] = 0.000000000000361 LocTimeSteps[1] = 0.000000000001083 dt_nyquist = 0.000000000166667 dt_sample = 0.000000000014079 tsPerSampling = 13 Number of samplings = 569 ========================================== total used free shared buff/cache available Mem: 128643 23485 64358 7670 40799 96266 Swap: 2047 0 2047 =================================================== Local Time-Stepping Loop =================================================== TIMER:: Start Time Stepping took 101 milliseconds E field norm^2 0.000000000000000  Current Time : 0.000000ns  Average iteration time : 101.000000 msec --------------------------------------------------- TIMER:: 13 steps took 1302 milliseconds E field norm^2 0.000000000000000  Current Time : 0.014079ns  Average iteration time : 100.214286 msec --------------------------------------------------- TIMER:: 13 steps took 1300 milliseconds E field norm^2 0.000000000000000  Current Time : 0.028159ns  Average iteration time : 100.111111 msec --------------------------------------------------- TIMER:: 13 steps took 1303 milliseconds E field norm^2 0.000000000000000  Current Time : 0.042238ns  Average iteration time : 100.150000 msec --------------------------------------------------- TIMER:: 13 steps took 1305 milliseconds E field norm^2 0.000000000000000  Current Time : 0.056318ns  Average iteration time : 100.207547 msec --------------------------------------------------- TIMER:: 13 steps took 1304 milliseconds E field norm^2 0.000000000000000  Current Time : 0.070397ns  Average iteration time : 100.227273 msec --------------------------------------------------- TIMER:: 13 steps took 1305 milliseconds E field norm^2 0.000000000000000  Current Time : 0.084477ns  Average iteration time : 100.253165 msec --------------------------------------------------- TIMER:: 13 steps took 1306 milliseconds E field norm^2 0.000000000000000  Current Time : 0.098556ns  Average iteration time : 100.282609 msec --------------------------------------------------- TIMER:: 13 steps took 1304 milliseconds E field norm^2 0.000000000000000  Current Time : 0.112636ns  Average iteration time : 100.285714 msec --------------------------------------------------- TIMER:: 13 steps took 1305 milliseconds E field norm^2 0.000000000000000  Current Time : 0.126715ns  Average iteration time : 100.296610 msec --------------------------------------------------- TIMER:: 13 steps took 1306 milliseconds E field norm^2 0.000000000000000  Current Time : 0.140795ns  Average iteration time : 100.312977 msec --------------------------------------------------- TIMER:: 13 steps took 1304 milliseconds E field norm^2 0.000000000000000  Current Time : 0.154874ns  Average iteration time : 100.312500 msec --------------------------------------------------- TIMER:: 13 steps took 1305 milliseconds E field norm^2 0.000000000000000  Current Time : 0.168954ns  Average iteration time : 100.318471 msec --------------------------------------------------- TIMER:: 13 steps took 1303 milliseconds E field norm^2 0.000000000000000  Current Time : 0.183033ns  Average iteration time : 100.311765 msec --------------------------------------------------- TIMER:: 13 steps took 1302 milliseconds E field norm^2 0.000000000000000  Current Time : 0.197112ns  Average iteration time : 100.300546 msec --------------------------------------------------- TIMER:: 13 steps took 1300 milliseconds E field norm^2 0.000000000000000  Current Time : 0.211192ns  Average iteration time : 100.280612 msec --------------------------------------------------- TIMER:: 13 steps took 1300 milliseconds E field norm^2 0.000000000000000  Current Time : 0.225271ns  Average iteration time : 100.263158 msec --------------------------------------------------- TIMER:: 13 steps took 1301 milliseconds E field norm^2 0.000000000000000  Current Time : 0.239351ns  Average iteration time : 100.252252 msec --------------------------------------------------- TIMER:: 13 steps took 1299 milliseconds E field norm^2 0.000000000000000  Current Time : 0.253430ns  Average iteration time : 100.234043 msec --------------------------------------------------- TIMER:: 13 steps took 1299 milliseconds E field norm^2 0.000000000000000  Current Time : 0.267510ns  Average iteration time : 100.217742 msec --------------------------------------------------- TIMER:: 13 steps took 1300 milliseconds E field norm^2 0.000000000000000  Current Time : 0.281589ns  Average iteration time : 100.206897 msec --------------------------------------------------- TIMER:: 13 steps took 1301 milliseconds E field norm^2 0.000000000000000  Current Time : 0.295669ns  Average iteration time : 100.200730 msec --------------------------------------------------- TIMER:: 13 steps took 1299 milliseconds E field norm^2 0.000000000000000  Current Time : 0.309748ns  Average iteration time : 100.188153 msec --------------------------------------------------- TIMER:: 13 steps took 1300 milliseconds E field norm^2 0.000000000000000  Current Time : 0.323828ns  Average iteration time : 100.180000 msec --------------------------------------------------- TIMER:: 13 steps took 1302 milliseconds E field norm^2 0.000000000000000  Current Time : 0.337907ns  Average iteration time : 100.178914 msec --------------------------------------------------- TIMER:: 13 steps took 1303 milliseconds E field norm^2 0.000000000000000  Current Time : 0.351987ns  Average iteration time : 100.180982 msec --------------------------------------------------- TIMER:: 13 steps took 1302 milliseconds E field norm^2 0.000000000000000  Current Time : 0.366066ns  Average iteration time : 100.179941 msec --------------------------------------------------- TIMER:: 13 steps took 1301 milliseconds E field norm^2 0.000000000000000  Current Time : 0.380145ns  Average iteration time : 100.176136 msec --------------------------------------------------- TIMER:: 13 steps took 1300 milliseconds E field norm^2 0.000000000000000  Current Time : 0.394225ns  Average iteration time : 100.169863 msec --------------------------------------------------- TIMER:: 13 steps took 1301 milliseconds E field norm^2 0.000000000000000  Current Time : 0.408304ns  Average iteration time : 100.166667 msec --------------------------------------------------- TIMER:: 13 steps took 1299 milliseconds E field norm^2 0.000000000000000  Current Time : 0.422384ns  Average iteration time : 100.158568 msec --------------------------------------------------- TIMER:: 13 steps took 1299 milliseconds E field norm^2 0.000000000000001  Current Time : 0.436463ns  Average iteration time : 100.150990 msec --------------------------------------------------- TIMER:: 13 steps took 1300 milliseconds E field norm^2 0.000000000000001  Current Time : 0.450543ns  Average iteration time : 100.146283 msec --------------------------------------------------- TIMER:: 13 steps took 1301 milliseconds E field norm^2 0.000000000000003  Current Time : 0.464622ns  Average iteration time : 100.144186 msec --------------------------------------------------- TIMER:: 13 steps took 1300 milliseconds E field norm^2 0.000000000000007  Current Time : 0.478702ns  Average iteration time : 100.139955 msec --------------------------------------------------- TIMER:: 13 steps took 1302 milliseconds E field norm^2 0.000000000000013  Current Time : 0.492781ns  Average iteration time : 100.140351 msec --------------------------------------------------- TIMER:: 13 steps took 1303 milliseconds E field norm^2 0.000000000000024  Current Time : 0.506861ns  Average iteration time : 100.142857 msec --------------------------------------------------- TIMER:: 13 steps took 1304 milliseconds E field norm^2 0.000000000000037  Current Time : 0.520940ns  Average iteration time : 100.147303 msec --------------------------------------------------- TIMER:: 13 steps took 1304 milliseconds E field norm^2 0.000000000000053  Current Time : 0.535020ns  Average iteration time : 100.151515 msec --------------------------------------------------- TIMER:: 13 steps took 1305 milliseconds E field norm^2 0.000000000000067  Current Time : 0.549099ns  Average iteration time : 100.157480 msec --------------------------------------------------- TIMER:: 13 steps took 1305 milliseconds E field norm^2 0.000000000000077  Current Time : 0.563179ns  Average iteration time : 100.163148 msec --------------------------------------------------- TIMER:: 13 steps took 1306 milliseconds