Dear Dr. Sreenath,
Thanks for sending the files. But I can see from your private message that you switched from the Del2a algorithm to Del2c and got it working. That is was I would
suggest. I’m glad you’re up and running!
Two of the biggest general challenges with this tool are its memory constraints and the processing time required to complete the calculations. With modern computers,
the memory constraint is largely artificial—a consequence of us only compiling the code into a 32-bit binary. On Windows, this limits the effective memory available to between 1 and 2 GB. Eventually, we plan to recompile it as a 64-bit binary which would allow
a virtually unlimited amount of memory to be utilized. But this brings us to the second problem: processing time required to run the simulation. Roughly speaking, both the memory utilization and the processing time grow with the square of the cell size. So
if you double the cell size, memory and processing time will increase by at least 4x. (It is actually worse than that, because as you decrease cell size, you must reduce the time step of the simulation so that calculations remain numerically stable. This results
in more time steps per day simulated.) Thus, if we unlimited the memory utilization (e.g. by creating a 64-bit version), it just pushes the problem to the next stage: it would allow small cell-size simulations to be created, but they would take forever to
run.
My colleague, Eric Treml (Deakin University), has made progress developing new code that can do parallel processing and take advantage of GPUs for some matrix
operations. If you are not able to progress with the current implementation in MGET, we could contact him and see if he had a better solution ready.
Best,
Jason
From: Sreenath Ramanathan <>
Sent: Tuesday, July 31, 2018 8:06 PM
To:
Cc: Jason Roberts <>
Subject: Re: [mget-help] Resampling error-MGET
I have made 3 rasters with WGS_1984_World_Mercator Projection and having common cell sizes. I could get an error again in the step 2 and I couldn't understand where I have
missed this time. As a trial I have included only 4 patches. I am sending the simulation directory (a bit heavy) in gdrive for your reference.
Executing: LarvalDispersalLoadHYCOMGLBa0084DEquatorialCurrentsIntoSimulation C:\Users\DELL\Documents\ArcGIS\Simulation5 20-09-2016 24-09-2016 0 # Degrees false CUBIC Del2a
60 120 C:\Users\DELL\Documents\ArcGIS\sim3Cache
Start Time: Wed Aug 01 04:37:41 2018
Running script LarvalDispersalLoadHYCOMGLBa0084DEquatorialCurrentsIntoSimulation...
Querying time and depth slices of u grid of the equatorial region of the HYCOM + NCODA Global 1/12 degree Analysis (GLBa0.08), clipped to indices tMin = 2924, tMax = 2928,
zMax = 0.
Query complete: 0:00:00 elapsed, 5 datasets found, 0:00:00.001000 per dataset.
Importing 5 datasets into ArcGIS Folder C:\Users\DELL\Documents\ArcGIS\Simulation5\OriginalHYCOMCurrents with mode "add".
Checking for existing destination datasets.
Finished checking: 0:00:00 elapsed, 5 datasets checked, 0:00:00.000600 per dataset.
0 destination datasets already exist. Importing 5 datasets.
Import in progress: 0:01:12 elapsed, 1 datasets imported, 0:01:12.175272 per dataset, 4 remaining, estimated completion time: unknown; more progress needed.
Import in progress: 0:02:43 elapsed, 3 datasets imported, 0:00:54.535062 per dataset, 2 remaining, estimated completion time: 04:41:22.
Import complete: 0:04:28 elapsed, 5 datasets imported, 0:00:53.791800 per dataset.
Querying time and depth slices of v grid of the equatorial region of the HYCOM + NCODA Global 1/12 degree Analysis (GLBa0.08), clipped to indices tMin = 2924, tMax = 2928,
zMax = 0.
Query complete: 0:00:00 elapsed, 5 datasets found, 0:00:00 per dataset.
Importing 5 datasets into ArcGIS Folder C:\Users\DELL\Documents\ArcGIS\Simulation5\OriginalHYCOMCurrents with mode "add".
Checking for existing destination datasets.
Finished checking: 0:00:00 elapsed, 5 datasets checked, 0:00:00 per dataset.
0 destination datasets already exist. Importing 5 datasets.
Import in progress: 0:01:34 elapsed, 2 datasets imported, 0:00:47.096269 per dataset, 3 remaining, estimated completion time: 04:46:21.
Import complete: 0:03:52 elapsed, 5 datasets imported, 0:00:46.444000 per dataset.
Finding ArcGIS rasters and inserting rows into table "work": workspace="C:\Users\DELL\Documents\ArcGIS\Simulation5\OriginalHYCOMCurrents", wildcard="*.img", searchTree=True,
rasterType="None"
Finished inserting rows: 0:00:00 elapsed, 20 rows inserted, 0:00:00.017500 per row.
Updating field "outputRaster" in table "work"...
Update complete: 0:00:00 elapsed, 20 rows updated, 0 deleted, 0 unchanged, 0:00:00 per row.
Querying the table "work" to build lists of inputs and outputs...
Query complete: 0:00:00 elapsed, 20 rows retrieved, 0:00:00 per row.
Checking for existing outputs for each input...
Finished checking for existing outputs: 0:00:00 elapsed, 20 inputs checked, 0:00:00.029350 per input.
RuntimeError: ??? Error using ==> mldivide
python.exe: RuntimeError: ??? Error using ==> mldivide
python.exe: Out of memory. Type HELP MEMORY for your options.
python.exe: Error in ==> InpaintNaNs>inpaint_nans at 252
python.exe: Error in ==> InpaintNaNs at 29
RuntimeError: python.exe returned exit code 1, indicating failure.
The following consequences resulted from the original error:
Execution of C:\Python27\ArcGIS10.3\python.exe failed.
Failed script LarvalDispersalLoadHYCOMGLBa0084DEquatorialCurrentsIntoSimulation...
Traceback (most recent call last):
File "C:\Program Files\GeoEco\ArcGISToolbox\Scripts\LarvalDispersalLoadHYCOMGLBa0084DEquatorialCurrentsIntoSimulation.py", line 5, in <module>
ExecuteMethodFromCommandLineAsArcGISTool('GeoEco.Connectivity.LarvalDispersal', 'LarvalDispersal', 'LoadHYCOMGLBa0084DEquatorialCurrentsIntoSimulation')
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\ArcGISScripts.py", line 210, in ExecuteMethodFromCommandLineAsArcGISTool
exec sourceCode in globals(), locals()
File "<string>", line 1, in <module>
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Connectivity\LarvalDispersal.py", line 311, in LoadHYCOMGLBa0084DEquatorialCurrentsIntoSimulation
cls._FinishLoadingCurrents(simulationDirectory, scp, thisCurrentsProduct, u'Center', 86400, hycomCurrentsDirectory, resamplingTechnique, interpolationMethod)
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Connectivity\LarvalDispersal.py", line 402, in _FinishLoadingCurrents
File "<string>", line 7, in FindAndProjectRastersToTemplate
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\BatchProcessing.py", line 2137, in ExecuteFindAndProcessMethod
boundProcessTableMethod(*tuple(argList))
File "<string>", line 5, in ProjectToTemplateTable
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\BatchProcessing.py", line 1271, in ExecuteProcessTableMethod
results = boundProcessListMethod(*tuple(argList))
File "<string>", line 5, in ProjectToTemplateList
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\BatchProcessing.py", line 888, in ExecuteProcessListMethod
result = boundMethod(*tuple(argList))
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataManagement\ArcGISRasters.py", line 1892, in ProjectToTemplate
Interpolator.InpaintArcGISRaster(projectedRaster, infilledRaster, interpolationMethod, maxHoleSize, minValue=minValue, maxValue=maxValue)
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\SpatialAnalysis\Interpolation.py", line 726, in InpaintArcGISRaster
ArcGISWorkspace(os.path.dirname(outputRaster), ArcGISRaster, pathCreationExpressions=[os.path.basename(outputRaster)]).ImportDatasets([InpaintedGrid(ArcGISRasterBand.ConstructFromArcGISPath(inputRaster),
method, maxHoleSize, xEdgesWrap, minValue, maxValue)], {False: u'Add', True: u'Replace'}[overwriteExisting], reportProgress=False)
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 820, in ImportDatasets
self._ImportDatasets(datasets, mode.lower(), reportProgress, options)
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Collections.py", line 695, in _ImportDatasets
self._ImportDatasetsToPath(pathComponentsForPath[path], datasetsForPath[path], mode, progressReporter, options)
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\ArcGIS.py", line 609, in _ImportDatasetsToPath
self.DatasetType._ImportDatasetsToPath(os.path.join(self.Path, *pathComponents), sourceDatasets, mode, progressReporter, options)
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\ArcGIS.py", line 1128, in _ImportDatasetsToPath
GDALDataset._ImportDatasetsToPath(path, sourceDatasets, mode, None, {'useArcGISSpatialReference': True, 'useUnscaledData': useUnscaledData, 'calculateStatistics': False,
'blockSize': blockSize})
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\GDAL.py", line 1073, in _ImportDatasetsToPath
data = "sourceDatasets[i].Data[rowsCopied:rowsCopied+rowsToCopy," :]
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 3671, in __getitem__
return getattr(self._Grid(), self._GetMethod)(key)
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 3311, in _GetUnscaledDataAsArray
data, actualNoDataValue = self._ReadNumpyArray(reorderedSliceList)
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Virtual.py", line 3600, in _ReadNumpyArray
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataManagement\Processes.py", line 575, in ExecuteProgram
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataManagement\Processes.py", line 498, in Execute
Logger.RaiseException(RuntimeError(_(u'%(name)s returned exit code %(code)i, indicating failure.') % {u'name' : programName, u'code' : self.ExitCode}))
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Logging.py", line 104, in RaiseException
RuntimeError: python.exe returned exit code 1, indicating failure.
Failed to execute (LarvalDispersalLoadHYCOMGLBa0084DEquatorialCurrentsIntoSimulation).
Failed at Wed Aug 01 04:47:03 2018 (Elapsed Time: 9 minutes 21 seconds)
|
![]()
|
|
SREENATH, K. R.
|
|
Scientist
|
|
Marine Biodiversity Division, ICAR-Central Marine Fisheries Research Institute
|
|
p:
|
(+91)484-2394867 m: (+91)9662212556
|
|
f:
|
(+91) 484 - 2394909
|
|
a:
|
CMFRI, Abraham Madamackal Road, Kochi, Kerala, India - 682 018.
|
|
w:
|
www.cmfri.org.in e:
|
|
![]()
|
|
On Wed, Aug 1, 2018 at 3:19 AM, Sreenath Ramanathan <> wrote:
I could recognize my mistake a little while before you have replied as I was checking the cell size of the images of current in 'u' folder in the sim dir. Perhaps the polygons
which I have used to make the patch ID created the trouble. Thanks for your fast reply. I am not sending you my Directory at this point, but will try to clear this error myself and get back to you if I again stumble up on any more errors. Again thanks for
spending your valuable time on this.
|
![]()
|
|
SREENATH, K. R.
|
|
Scientist
|
|
Marine Biodiversity Division, ICAR-Central Marine Fisheries Research Institute
|
|
p:
|
(+91)484-2394867 m: (+91)9662212556
|
|
f:
|
(+91) 484 - 2394909
|
|
a:
|
CMFRI, Abraham Madamackal Road, Kochi, Kerala, India - 682 018.
|
|
w:
|
www.cmfri.org.in e:
|
|
![]()
|
|
On Wed, Aug 1, 2018 at 3:09 AM, Jason Roberts <> wrote:
Dear Dr. Sreenath,
What is the coordinate system used by your rasters? And what is the cell size?
In order to use the larval dispersal tool, it is necessary that your rasters be in a projected coordinate
system. Such a system will use a linear unit of measure, typically meters. It is necessary to use a projected system so that physical calculations are easy to perform.
When you say you used patches with cell sizes of 0.08 and 0.046, it sounds like those units are in
degrees, which is an angular unit. This suggests that the rasters are not in a projected coordinate system. The Create Larval Connectivity Simulation tool is supposed to fail if you provide rasters in an unprojected coordinate system. Did that happen?
Alternatively, it may be that you created rasters in a projected system with cell sizes of 0.08 or
0.046. This seems very unlikely, because those sizes would be in meters, resulting in a raster with a huge number of rows and columns. But if that is indeed what happened, it is not surprising that the Resampling tool used inside MGET would fail.
If you would like, you may send me the Simulation Directory created by the Create Larval Connectivity
Simulation tool. I will check it to see if the coordinate system and the cell size seems reasonable for use with the HYCOM GLBa0.08 data.
Best regards,
Jason
From:
<>
On Behalf Of Sreenath Ramanathan
Sent: Tuesday, July 31, 2018 3:10 PM
To:
Subject: [mget-help] Resampling error-MGET
I got Resampling failed error 000995 when I used patches of both the cellsizes 0.08 and 0.046. I was trying to
simulate the coral larvae of eastern part of Arabian Sea using default GLBa0.08. Patch cover, ID and Wmask have same cellsize and row and columns.
|
![]()
|
|
SREENATH, K. R.
|
|
Scientist
|
|
Marine Biodiversity Division, ICAR-Central Marine Fisheries Research Institute
|
|
p:
|
(+91)484-2394867 m: (+91)9662212556
|
|
f:
|
(+91) 484 - 2394909
|
|
a:
|
CMFRI, Abraham Madamackal Road, Kochi, Kerala, India - 682 018.
|
|
w:
|
www.cmfri.org.in e:
|
|
![]()
|
|
|