Subject: Marine Geospatial Ecology Tools (MGET) help
Text archives
| From: | Jason Roberts <> |
|---|---|
| To: | Bryan Costa - NOAA Affiliate <> |
| Cc: | "" <> |
| Subject: | RE: [mget-help] Problem using Find ArcGIS Rasters and Interpolate No Data Cells tool |
| Date: | Fri, 26 Sep 2014 17:38:03 +0000 |
| Accept-language: | en-US |
|
Hi Bryan, Thanks for your interest in MGET. Is it correct that your raster has dimensions of 37912 x 35946? If so, I’m afraid you are hitting a fundamental limitation
of this tool, and there is no easy workaround. Normally, for rasters of this size, it is necessary to process them in chunks using a moving window because the entire raster cannot fit in memory. Unfortunately
the algorithm used by this tool is not easily adaptable to that approach. To fill a given No Data cell, the algorithm relies on values from surrounding cells from all sides, extending out several cells. (I do not know exactly how far; I did not write the math
that is the core of the algorithm.) So it is not sufficient to do it with a moving window that moves by the entire width of the window at each step. This would result in “edge effects” at the edges of each window, because the No Data cells there would only
consider those within the window, not outside it. To do it properly, there needs to be a buffer of several cells around the window. That is somewhat complicated to implement, and I did not have time to do it.
As a result, the tool tries to load the entire raster into memory and process it at one go. For your raster, about 225 million cells (1 GB of 32-bit integer values), there’s not enough memory. In theory, there could be enough memory, but in practice, the 32-bit
address space in which the algorithm runs will never have it, due to heap fragmentation and limitations imposed by the operating system. The only workaround I can suggest at this time is to split your raster into tiles yourself, process each tile individually, and then mosaic them back together.
That will be tedious, and there will be edge effects along the tile edges unless you buffer as I described. I’m not sure how bad those effects would be; it depends on the data. Sorry I can’t be of more help, Jason From: Bryan Costa - NOAA Affiliate [mailto:]
Hi, I'm writing because I keep getting an error when I run MGET's Find ArcGIS Rasters and Interpolate No Data Cells tool. The error is listed below. I tried reducing the size of the rasters (because it says the array is too big), but still
no luck. I look forward to hearing from you at your convenience. Regards, Bryan C --------------------------------------------- Error Executing: InterpolatorFindAndInpaintArcGISRasters F:\Model_Predictors\Resolution_1p2x1p2km F:\Model_Predictors\Fill_Holidays Del2a # false * false # os.path.join(outputWorkspace, inputRaster[len(workspaceToSearch)+1:]) os.path false Start Time: Fri Sep 26 12:39:42 2014 Running script InterpolatorFindAndInpaintArcGISRasters... Finding ArcGIS rasters and inserting rows into table "work": workspace="F:\Model_Predictors\Resolution_1p2x1p2km", wildcard="*", searchTree=False, rasterType="None" Finished inserting rows: 0:00:00 elapsed, 48 rows inserted, 0:00:00.002750 per row. Updating field "outputRaster" in table "work"... Update complete: 0:00:00 elapsed, 48 rows updated, 0 deleted, 0 unchanged, 0:00:00.000270 per row. Querying the table "work" to build lists of inputs and outputs... Query complete: 0:00:00 elapsed, 48 rows retrieved, 0:00:00.000166 per row. Processing 48 inputs... RuntimeError: Failed to retrieve a block of data of 37912 columns by 35946 rows at offsets x=0, y=0 from band 1 of band 1 of GDAL dataset "F:\Model_Predictors\Resolution_1p2x1p2km\Chlorophylla_01Jan_Mean_2003to2013_1p2x1p2km.tif" with the
Geospatial Data Abstraction Library (GDAL). Verify that the dataset exists, is accessible, and has the expected dimensions. Detailed error information: band.ReadAsArray(0, 0, 37912, 35946) reported ValueError: array is too big.. Traceback (most recent call last): File "C:\Program Files\GeoEco\ArcGISToolbox\Scripts\InterpolatorFindAndInpaintArcGISRasters.py", line 5, in <module> ExecuteMethodFromCommandLineAsArcGISTool('GeoEco.SpatialAnalysis.Interpolation', 'Interpolator', 'FindAndInpaintArcGISRasters') File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\ArcGISScripts.py", line 210, in ExecuteMethodFromCommandLineAsArcGISTool exec sourceCode in globals(), locals() File "<string>", line 1, in <module> File "<string>", line 7, in FindAndInpaintArcGISRasters File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\BatchProcessing.py", line 2161, in ExecuteFindAndProcessMethod boundProcessTableMethod(*tuple(argList)) File "<string>", line 5, in InpaintArcGISRasterTable File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\BatchProcessing.py", line 1295, in ExecuteProcessTableMethod results = boundProcessListMethod(*tuple(argList)) File "<string>", line 5, in InpaintArcGISRasterList File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\BatchProcessing.py", line 912, in ExecuteProcessListMethod result = boundMethod(*tuple(argList)) File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\SpatialAnalysis\Interpolation.py", line 726, in InpaintArcGISRaster ArcGISWorkspace(os.path.dirname(outputRaster), ArcGISRaster, pathCreationExpressions=[os.path.basename(outputRaster)]).ImportDatasets([InpaintedGrid(ArcGISRasterBand.ConstructFromArcGISPath(inputRaster), method, maxHoleSize, xEdgesWrap)],
{False: u'Add', True: u'Replace'}[overwriteExisting], reportProgress=False) File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\Datasets\__init__.py", line 816, in ImportDatasets self._ImportDatasets(datasets, mode.lower(), reportProgress, options) File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\Datasets\Collections.py", line 694, in _ImportDatasets self._ImportDatasetsToPath(pathComponentsForPath[path], datasetsForPath[path], mode, progressReporter, options) File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\Datasets\ArcGIS.py", line 607, in _ImportDatasetsToPath self.DatasetType._ImportDatasetsToPath(os.path.join(self.Path, *pathComponents), sourceDatasets, mode, progressReporter, options) File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\Datasets\ArcGIS.py", line 1119, in _ImportDatasetsToPath GDALDataset._ImportDatasetsToPath(path, sourceDatasets, mode, None, {'useArcGISSpatialReference': True, 'useUnscaledData': useUnscaledData, 'calculateStatistics': False, 'blockSize': blockSize}) File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\Datasets\GDAL.py", line 1073, in _ImportDatasetsToPath data = "sourceDatasets[i].Data[rowsCopied:rowsCopied+rowsToCopy," :] File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\Datasets\__init__.py", line 3666, in __getitem__ return getattr(self._Grid(), self._GetMethod)(key) File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\Datasets\__init__.py", line 3306, in _GetUnscaledDataAsArray data, actualNoDataValue = self._ReadNumpyArray(reorderedSliceList) File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\Datasets\Virtual.py", line 3337, in _ReadNumpyArray grid.Data[:].tofile(holesFile) File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\Datasets\__init__.py", line 3666, in __getitem__ return getattr(self._Grid(), self._GetMethod)(key) File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\Datasets\__init__.py", line 3306, in _GetUnscaledDataAsArray data, actualNoDataValue = self._ReadNumpyArray(reorderedSliceList) File "C:\Python27\ArcGIS10.1\lib\site-packages\GeoEco\Datasets\GDAL.py", line 1374, in _ReadNumpyArray raise RuntimeError(_(u'Failed to retrieve a block of data of %(win_xsize)i columns by %(win_ysize)i rows at offsets x=%(xoff)i, y=%(yoff)i from band %(band)i of %(dn)s with the Geospatial Data Abstraction Library (GDAL). Verify that
the dataset exists, is accessible, and has the expected dimensions. Detailed error information: band.ReadAsArray(%(xoff)i, %(yoff)i, %(win_xsize)i, %(win_ysize)i) reported %(e)s: %(msg)s.') % {u'band': self.Band, u'dn': self.DisplayName, u'xoff': xoff, u'yoff':
yoff, u'win_xsize': win_xsize, u'win_ysize': win_ysize, u'e': e.__class__.__name__, u'msg': self._Unicode(e)}) RuntimeError: Failed to retrieve a block of data of 37912 columns by 35946 rows at offsets x=0, y=0 from band 1 of band 1 of GDAL dataset "F:\Model_Predictors\Resolution_1p2x1p2km\Chlorophylla_01Jan_Mean_2003to2013_1p2x1p2km.tif" with the
Geospatial Data Abstraction Library (GDAL). Verify that the dataset exists, is accessible, and has the expected dimensions. Detailed error information: band.ReadAsArray(0, 0, 37912, 35946) reported ValueError: array is too big.. Failed to execute (InterpolatorFindAndInpaintArcGISRasters). Failed at Fri Sep 26 12:39:50 2014 (Elapsed Time: 8.00 seconds) -- Bryan Costa Geospatial Scientist Ecosystem Modeler CSS-Dynamac NOAA|CCMA|Biogeography Branch 1305 East West Highway N-SCI-1, SSMC 4, 9th Floor, #9232 Silver Spring, MD 20910 Phone: (301) 713-3028 x146 Fax: (301) 713-4384 Email: The contents of this message are mine and do not necessarily reflect any position of NOAA |