Subject: Marine Geospatial Ecology Tools (MGET) help
Text archives
From: | "Jason Roberts" <> |
---|---|
To: | "Havik GJ" <>, |
Cc: | "Eijnsbergen C van" <> |
Subject: | Re: [mget-help] Filling NoData failed |
Date: | Sat, 17 Aug 2013 15:03:48 -0400 |
Dear GJ,
I'm sorry about this problem. I should be able to help you fix it but am on holiday so my time and resources are limited. I'll be back Sept 2 and can look at it more deeply at that time.
In the mean time, I have several thoughts. First, I think the root problem is the KeyError: u'band'. The sequence of events probably is:
1. The KeyError occurs but the message is not reported for it yet.
2. The tool enters its error handling logic and tries to clean up temporary files. But the root failure occurred in a low level library, and it left some files open. The tool fails to delete those files and issues the Error 32 warnings.
3. The clean up code finishes. The tool now reports the KeyError and aborts execution.
To solve this, we must find out what is causing the KeyError. At first glance, that looks like a bug in MGET, although I am not sure where. I have these suggestions:
1. This tool requires a lot of memory. The most frequent failure that users encounter occurs when they try to use it on a large raster. Unfortunately I have not characterized it so I cannot say how large a raster will cause the memory failure. When this happens, usually there is a message indicating a memory problem. To test whether this is happening to you, clip your raster down to something very small, such as 200 x 200 cells and see if you can get it working on that. If so, perhaps memory is the problem. On the large raster, try the different algorithms to see if one will use less memory than another. I think the Del2c uses least memory, but I can't remember without looking at the code. Also, please let me know how big your raster is.
2. If you cannot get it working even on a small raster, memory is probably not the issue. My second idea is that the problem relates to the raster being in a geodatabase. This is where I the bug in MGET is most likely to be, if there is a bug. MGET cannot read rasters directly from geodatabases directly using an ESRI API, at least not in 9.3.1, because ESRI did not provide an API for it in that release. Instead MGET copies the raster out of the geodatabase and uses the GDAL library to read it. Perhaps MGET is not doing this; perhaps it is trying to read the raster directly from the geodatabase using GDAL, and that is failing in an odd way. (GDAL does not support reading rasters from geodatabases, but perhaps it is trying to do so and giving this odd error message.) To test whether this is happening, copy the raster out of the geodatabase and try the tool on the copy. If this works, and the bug is that MGET is not doing this itself, it should be easy for me to fix. Of course, the Error 32 messages suggest that MGET has already made a temporary copy of the geodatabase raster, which would suggest that this is not the problem...
3. I can better understand the problem if I can see some debugging information. Please enable verbose logging, reproduce the problem, and send me the full output.
To use this from a Python script, you can simply do this:
from GeoEco.SpatialAnalysis.Interpolation import Interpolator
Interpolator.InpaintArcGISRaster(inputRaster, outputRaster)
See here for documentation (this is also available on your local machine through the Windows Start menu). Note that MGET does not provide a formal API at this time. You are welcome to call any functions in MGET, of course, but it is possible that any of them may change in the future. We hope to do a better job eventually.
Also, it is possible to call the tool through the ArcGIS geoprocessor, like other ArcGIS tools, but I do not recommend that. The MGET toolbox is so large that ArcGIS has a hard time using it through their normal mechanism (although I don't think these problems started until ArcGIS 10.0). We are working with ESRI to solve this problem. Until it is solved, it is just easier to call MGET like any other Python module, rather than through the geoprocessor
Best regards,
Jason
----- Original Message -----From:"Havik GJ" @pzh.nl>To:""Cc:"Eijnsbergen C van"Sent:Fri, 16 Aug 2013 12:02:08 +0200Subject:[mget-help] Filling NoData failed@pzh.nl>@nicholas.duke.edu>Greetings,
Today we used the new version of MGET to fill in the holes of a raster. After a few minutes interpolation stopped.
We use ArcGis 9.3.1 on Windows 7. Installed (as far as we know) all the required components.
This is what happened.
Executing: InterpolatorInpaintArcGISRaster D:\DATA\AHN2_CORRECT\AHN2.gdb\AHN2_5MGRID D:\DATA\AHN2_CORRECT\AHN2_GEVULD.gdb\AHN2_5M Del2a # false
Start Time: Fri Aug 16 11:43:10 2013
Running script InterpolatorInpaintArcGISRaster...
Could not delete temporary file c:\users\havikgj\appdata\local\temp\GeoEco_ArcGISRaster_Temp_pift0r\raster.img due to WindowsError: [Error 32] Het proces heeft geen toegang tot het bestand omdat het door een ander
proces wordt gebruikt: 'c:\\users\\havikgj\\appdata\\local\\temp\\GeoEco_ArcGISRaster_Temp_pift0r\\raster.img'
Could not delete temporary file c:\users\havikgj\appdata\local\temp\GeoEco_ArcGISRaster_Temp_pift0r\raster.rrd due to WindowsError: [Error 32] Het proces heeft geen toegang tot het bestand omdat het door een ander
proces wordt gebruikt: 'c:\\users\\havikgj\\appdata\\local\\temp\\GeoEco_ArcGISRaster_Temp_pift0r\\raster.rrd'
Could not delete temporary directory c:\users\havikgj\appdata\local\temp\GeoEco_ArcGISRaster_Temp_pift0r due to WindowsError: [Error 145] De map is niet leeg: 'c:\\users\\havikgj\\appdata\\local\\temp\\GeoEco_ArcGISRaster_Temp_pift0r'
KeyError: u'band'
: u'band'
Failed to execute (InterpolatorInpaintArcGISRaster).
End Time: Fri Aug 16 11:45:50 2013 (Elapsed Time: 2 minutes 40 seconds)
The application in [Error 32]is AcrCatalog.
We tried in ArcCatalog and ArcMap, same result.
I wanted to try using it in a Pythonscript outside ArcGis with a faint hope that there would be no locking but could not figure out how to accomplish that. A sample script would be appreciated.
Do you have any other idea how to correct this?
Met vriendelijke groet,
G.J. (Gerard) Havik
Dataspecialist
Afdeling I&A | bureau Datamanagement
T 070 441 8796
Provincie Zuid-Holland | Zuid-Hollandplein 1
Postbus 90602 | 2509 LP Den Haag
-Wanneer u de provincie Zuid-Holland een e-mail stuurt, ontvangt u binnen 2 werkdagen een reactie en binnen twee weken een antwoord. Al uw informatie wordt vertrouwelijk behandeld
. Persoons- of adresgegevens worden uitsluitend gebruikt waarvoor u ze heeft verstrekt. Uw e-mailbericht wordt op een goede en veilige manier gearchiveerd.-Vragen kunt u stellen via het contactformulier.
P een gedachte voor het milieu: is printen van deze mail echt nodig?