Skip to Content.

mget-help - RE:[mget-help]

Please Wait...

Subject: Marine Geospatial Ecology Tools (MGET) help

Text archives


RE:[mget-help]


Chronological Thread 
  • From: Jason Roberts <>
  • To: "Warner, David" <>
  • Cc: "" <>
  • Subject: RE:[mget-help]
  • Date: Fri, 30 Oct 2015 15:48:26 +0000
  • Accept-language: en-US
  • Authentication-results: spf=none (sender IP is ) ;
  • Spamdiagnosticmetadata: NSPM
  • Spamdiagnosticoutput: 1:23

David,

 

Thanks for bringing this to our attention; it is a bug in MGET. NASA recently switched the ocean color data from HDF4 to netCDF4 format. Part of this change caused the product codes—RRS_Rrs_443 in your case—to become case sensitive. When you ran the tool, you specified the correct case but the tool improperly converted it to lower case at a step of the conversion process, causing it to fail.

 

I attached a patched file that will fix this problem for you. To apply it (assuming you have ArcGIS 10.3, which your output indicates, and MGET 0.8a60):

 

1.    Close all ArcGIS programs.

2.    Save the attached file to C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataProducts\NASA\OceanColor.py, overwriting the file that is there.

3.    Restart ArcGIS and try again.

 

Let me know how it goes.

 

Best,

Jason

 

From: [mailto:] On Behalf Of Warner, David
Sent: Thursday, October 29, 2015 1:47 PM
To:
Subject: [mget-help]

 

Greetings

 

I have been having some issues with mget that I've never had before.

 

I've been trying to download L3 SMI data from the oceancolor site using the data products Create Rasters for NASA OceanColor L3 SMI Product tool.

 

I get the result below at both daily and monthly temporal resolution and am not sure why.  I know the daily files exist on the NASA server.

 

Executing: OceanColorLevel3SMITimeSeriesCreateArcGISRasters Aqua Daily 4km RRS_Rrs_443 "G:\Lesht chla" Add %(Sensor)s;%(TemporalResolution)s;%(SpatialResolution)s;%(ProductCode)s;%%Y;%(SensorCode).1s%%Y%%j%(EndDate)s.L3m_%(TemporalResolutionCode).3s_%(ProductCode)s_%(SpatialResolution)s.img # # # "-92 41 -76 50" 1/1/2003 9/30/2015 6000000 3000000 true false

Start Time: Thu Oct 29 13:41:33 2015

Running script OceanColorLevel3SMITimeSeriesCreateArcGISRasters...

Querying time slices of aqua daily 4km rrs_rrs_443 from NASA OceanColor, clipped to indices tMin = 182, tMax = 4837, yMin = 3144, yMax = 3359, xMin = 2112, xMax = 2495.

Query complete: 0:00:06 elapsed, 4656 datasets found, 0:00:00.001405 per dataset.

Importing 4656 datasets into ArcGIS Folder G:\Lesht chla with mode "add".

Checking for existing destination datasets.

Finished checking: 0:00:02 elapsed, 4656 datasets checked, 0:00:00.000607 per dataset.

0 destination datasets already exist. Importing 4656 datasets.

Import stopped before all datasets were imported: 0:00:04 elapsed, 0 datasets imported, 0:00:00 per dataset, 4656 datasets not imported.

RuntimeError: Failed to open a variable named "rrs_443" in netCDF file c:\users\dmwarner\appdata\local\temp\1\arc1840\GeoEco_OceanColorLevel3SMIFileSearcher_Temp_ilsgac\A2003001.L3m_DAY_RRS_Rrs_443_4km.nc (a local copy of A2003001.L3m_DAY_RRS_Rrs_443_4km.nc from NASA OceanColor data archive). Detailed error information: KeyError: 'rrs_443'.

Failed script OceanColorLevel3SMITimeSeriesCreateArcGISRasters...

 

Traceback (most recent call last):

  File "C:\Program Files\GeoEco\ArcGISToolbox\Scripts\OceanColorLevel3SMITimeSeriesCreateArcGISRasters.py", line 5, in <module>

    ExecuteMethodFromCommandLineAsArcGISTool('GeoEco.DataProducts.NASA.OceanColor', 'OceanColorLevel3SMITimeSeries', 'CreateArcGISRasters')

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\ArcGISScripts.py", line 210, in ExecuteMethodFromCommandLineAsArcGISTool

    exec sourceCode in globals(), locals()

  File "<string>", line 1, in <module>

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataProducts\NASA\OceanColor.py", line 582, in CreateArcGISRasters

    workspace.ImportDatasets(GridSliceCollection(grid).QueryDatasets(), mode, calculateStatistics=calculateStatistics, buildPyramids=buildPyramids)

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 820, in ImportDatasets

    self._ImportDatasets(datasets, mode.lower(), reportProgress, options)

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Collections.py", line 694, in _ImportDatasets

    self._ImportDatasetsToPath(pathComponentsForPath[path], datasetsForPath[path], mode, progressReporter, options)

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\ArcGIS.py", line 609, in _ImportDatasetsToPath

    self.DatasetType._ImportDatasetsToPath(os.path.join(self.Path, *pathComponents), sourceDatasets, mode, progressReporter, options)

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\ArcGIS.py", line 1121, in _ImportDatasetsToPath

    GDALDataset._ImportDatasetsToPath(path, sourceDatasets, mode, None, {'useArcGISSpatialReference': True, 'useUnscaledData': useUnscaledData, 'calculateStatistics': False, 'blockSize': blockSize})

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\GDAL.py", line 1073, in _ImportDatasetsToPath

    data = "sourceDatasets[i].Data[rowsCopied:rowsCopied+rowsToCopy," :]

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 3670, in __getitem__

    return getattr(self._Grid(), self._GetMethod)(key)

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Virtual.py", line 562, in _GetUnscaledDataAsArray

    return self._Grid._GetUnscaledDataAsArray(self._AddSlicedDimsToKey(key))

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 3310, in _GetUnscaledDataAsArray

    data, actualNoDataValue = self._ReadNumpyArray(reorderedSliceList)

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Virtual.py", line 1058, in _ReadNumpyArray

    return self._Grid._ReadNumpyArray(slicesToRead)

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Virtual.py", line 340, in _ReadNumpyArray

    data[t] = self._CachedDatasets[i].UnscaledData.__getitem__(tuple(sliceList[1:]))

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 3670, in __getitem__

    return getattr(self._Grid(), self._GetMethod)(key)

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 3310, in _GetUnscaledDataAsArray

    data, actualNoDataValue = self._ReadNumpyArray(reorderedSliceList)

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\NetCDF.py", line 735, in _ReadNumpyArray

    v = self._GetVariable()

  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\NetCDF.py", line 726, in _GetVariable

    raise RuntimeError(_(u'Failed to open a variable named "%(name)s" in %(dn)s. Detailed error information: %(e)s: %(msg)s.') % {u'name': self._VariableName, u'dn': self.ParentCollection.DisplayName, u'e': e.__class__.__name__, u'msg': self._Str(e)})

RuntimeError: Failed to open a variable named "rrs_443" in netCDF file c:\users\dmwarner\appdata\local\temp\1\arc1840\GeoEco_OceanColorLevel3SMIFileSearcher_Temp_ilsgac\A2003001.L3m_DAY_RRS_Rrs_443_4km.nc (a local copy of A2003001.L3m_DAY_RRS_Rrs_443_4km.nc from NASA OceanColor data archive). Detailed error information: KeyError: 'rrs_443'.

 

Failed to execute (OceanColorLevel3SMITimeSeriesCreateArcGISRasters).

Failed at Thu Oct 29 13:41:57 2015 (Elapsed Time: 24.19 seconds)

 

 

--

David Warner

Research Fisheries Biologist

U.S.G.S. Great Lakes Science Center

1451 Green Road

Ann Arbor, MI 48105

734-214-9392 

Attachment: OceanColor.py
Description: OceanColor.py



  • [mget-help], Warner, David, 10/29/2015
    • RE:[mget-help], Jason Roberts, 10/30/2015

Archive powered by MHonArc 2.6.19.

Top of Page