Subject: Marine Geospatial Ecology Tools (MGET) help
Text archives
- From: Jason Roberts <>
- To: "Warner, David" <>
- Cc: "" <>
- Subject: RE:[mget-help]
- Date: Fri, 30 Oct 2015 15:48:26 +0000
- Accept-language: en-US
- Authentication-results: spf=none (sender IP is ) ;
- Spamdiagnosticmetadata: NSPM
- Spamdiagnosticoutput: 1:23
David,
Thanks for bringing this to our attention; it is a bug in MGET. NASA recently switched the ocean color data from HDF4 to netCDF4 format. Part of this change caused the product codes—RRS_Rrs_443 in your case—to become case sensitive. When you ran the tool, you specified the correct case but the tool improperly converted it to lower case at a step of the conversion process, causing it to fail.
I attached a patched file that will fix this problem for you. To apply it (assuming you have ArcGIS 10.3, which your output indicates, and MGET 0.8a60):
1. Close all ArcGIS programs. 2. Save the attached file to C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataProducts\NASA\OceanColor.py, overwriting the file that is there. 3. Restart ArcGIS and try again.
Let me know how it goes.
Best, Jason
From: [mailto:]
On Behalf Of Warner, David
Greetings
I have been having some issues with mget that I've never had before.
I've been trying to download L3 SMI data from the oceancolor site using the data products Create Rasters for NASA OceanColor L3 SMI Product tool.
I get the result below at both daily and monthly temporal resolution and am not sure why. I know the daily files exist on the NASA server.
Executing: OceanColorLevel3SMITimeSeriesCreateArcGISRasters Aqua Daily 4km RRS_Rrs_443 "G:\Lesht chla" Add %(Sensor)s;%(TemporalResolution)s;%(SpatialResolution)s;%(ProductCode)s;%%Y;%(SensorCode).1s%%Y%%j%(EndDate)s.L3m_%(TemporalResolutionCode).3s_%(ProductCode)s_%(SpatialResolution)s.img # # # "-92 41 -76 50" 1/1/2003 9/30/2015 6000000 3000000 true false Start Time: Thu Oct 29 13:41:33 2015 Running script OceanColorLevel3SMITimeSeriesCreateArcGISRasters... Querying time slices of aqua daily 4km rrs_rrs_443 from NASA OceanColor, clipped to indices tMin = 182, tMax = 4837, yMin = 3144, yMax = 3359, xMin = 2112, xMax = 2495. Query complete: 0:00:06 elapsed, 4656 datasets found, 0:00:00.001405 per dataset. Importing 4656 datasets into ArcGIS Folder G:\Lesht chla with mode "add". Checking for existing destination datasets. Finished checking: 0:00:02 elapsed, 4656 datasets checked, 0:00:00.000607 per dataset. 0 destination datasets already exist. Importing 4656 datasets. Import stopped before all datasets were imported: 0:00:04 elapsed, 0 datasets imported, 0:00:00 per dataset, 4656 datasets not imported. RuntimeError: Failed to open a variable named "rrs_443" in netCDF file c:\users\dmwarner\appdata\local\temp\1\arc1840\GeoEco_OceanColorLevel3SMIFileSearcher_Temp_ilsgac\A2003001.L3m_DAY_RRS_Rrs_443_4km.nc (a local copy of A2003001.L3m_DAY_RRS_Rrs_443_4km.nc from NASA OceanColor data archive). Detailed error information: KeyError: 'rrs_443'. Failed script OceanColorLevel3SMITimeSeriesCreateArcGISRasters...
Traceback (most recent call last): File "C:\Program Files\GeoEco\ArcGISToolbox\Scripts\OceanColorLevel3SMITimeSeriesCreateArcGISRasters.py", line 5, in <module> ExecuteMethodFromCommandLineAsArcGISTool('GeoEco.DataProducts.NASA.OceanColor', 'OceanColorLevel3SMITimeSeries', 'CreateArcGISRasters') File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\ArcGISScripts.py", line 210, in ExecuteMethodFromCommandLineAsArcGISTool exec sourceCode in globals(), locals() File "<string>", line 1, in <module> File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataProducts\NASA\OceanColor.py", line 582, in CreateArcGISRasters workspace.ImportDatasets(GridSliceCollection(grid).QueryDatasets(), mode, calculateStatistics=calculateStatistics, buildPyramids=buildPyramids) File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 820, in ImportDatasets self._ImportDatasets(datasets, mode.lower(), reportProgress, options) File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Collections.py", line 694, in _ImportDatasets self._ImportDatasetsToPath(pathComponentsForPath[path], datasetsForPath[path], mode, progressReporter, options) File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\ArcGIS.py", line 609, in _ImportDatasetsToPath self.DatasetType._ImportDatasetsToPath(os.path.join(self.Path, *pathComponents), sourceDatasets, mode, progressReporter, options) File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\ArcGIS.py", line 1121, in _ImportDatasetsToPath GDALDataset._ImportDatasetsToPath(path, sourceDatasets, mode, None, {'useArcGISSpatialReference': True, 'useUnscaledData': useUnscaledData, 'calculateStatistics': False, 'blockSize': blockSize}) File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\GDAL.py", line 1073, in _ImportDatasetsToPath data = "sourceDatasets[i].Data[rowsCopied:rowsCopied+rowsToCopy," :] File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 3670, in __getitem__ return getattr(self._Grid(), self._GetMethod)(key) File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Virtual.py", line 562, in _GetUnscaledDataAsArray return self._Grid._GetUnscaledDataAsArray(self._AddSlicedDimsToKey(key)) File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 3310, in _GetUnscaledDataAsArray data, actualNoDataValue = self._ReadNumpyArray(reorderedSliceList) File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Virtual.py", line 1058, in _ReadNumpyArray return self._Grid._ReadNumpyArray(slicesToRead) File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Virtual.py", line 340, in _ReadNumpyArray data[t] = self._CachedDatasets[i].UnscaledData.__getitem__(tuple(sliceList[1:])) File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 3670, in __getitem__ return getattr(self._Grid(), self._GetMethod)(key) File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 3310, in _GetUnscaledDataAsArray data, actualNoDataValue = self._ReadNumpyArray(reorderedSliceList) File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\NetCDF.py", line 735, in _ReadNumpyArray v = self._GetVariable() File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\NetCDF.py", line 726, in _GetVariable raise RuntimeError(_(u'Failed to open a variable named "%(name)s" in %(dn)s. Detailed error information: %(e)s: %(msg)s.') % {u'name': self._VariableName, u'dn': self.ParentCollection.DisplayName, u'e': e.__class__.__name__, u'msg': self._Str(e)}) RuntimeError: Failed to open a variable named "rrs_443" in netCDF file c:\users\dmwarner\appdata\local\temp\1\arc1840\GeoEco_OceanColorLevel3SMIFileSearcher_Temp_ilsgac\A2003001.L3m_DAY_RRS_Rrs_443_4km.nc (a local copy of A2003001.L3m_DAY_RRS_Rrs_443_4km.nc from NASA OceanColor data archive). Detailed error information: KeyError: 'rrs_443'.
Failed to execute (OceanColorLevel3SMITimeSeriesCreateArcGISRasters). Failed at Thu Oct 29 13:41:57 2015 (Elapsed Time: 24.19 seconds)
-- David Warner Research Fisheries Biologist U.S.G.S. Great Lakes Science Center 1451 Green Road Ann Arbor, MI 48105 734-214-9392 |
Attachment:
OceanColor.py
Description: OceanColor.py
- [mget-help], Warner, David, 10/29/2015
- RE:[mget-help], Jason Roberts, 10/30/2015
Archive powered by MHonArc 2.6.19.