Skip to Content.

mget-help - RE: [mget-help] Bug with Hycom data

Please Wait...

Subject: Marine Geospatial Ecology Tools (MGET) help

Text archives


From: Jason Roberts <>
To: Grant Humphries <>
Cc: "" <>
Subject: RE: [mget-help] Bug with Hycom data
Date: Tue, 9 Jun 2015 21:01:48 +0000
Accept-language: en-US
Authentication-results: gmail.com; dkim=none (message not signed) header.d=none;

Grant,

 

Sorry for my slow reply. Thanks for the data. I took the initiative of not approving your message for the mget-help list, in case you preferred your data to remain confidential, but I’m CCing the list here in case anyone is interested in the outcome.

 

In short, the problem is 1) the HYCOM server is being flaky and not always responding, and 2) somehow set the value of the OPeNDAP options à Maximum Retry Time parameter to None (empty in the ArcGIS user interface) for some of the HYCOM tools. As a result, these tools do not retry any failures, so one small hiccup by the server and the whole run fails.

 

I have fixed #2 in the code but this does not help you. Here’s what you can do to make the tool faster and more resilient to these problems with the server:

 

1.    Under OPeNDAP options, set Maximum Retry Time to some large number, like 3600 seconds. This will cause the tool to retry failures for an hour before giving up.

 

2.    Also consider setting a value for Cache Directory. The tool will put data in here after retrieving it. Then, if it runs again, it will check this directory to see if it already retrieved it, and if so, get it from there rather than hitting the server.

 

3.    Under Interpolation options, set the Where Clause parameter to cur_speed <= 0. This will cause the tool to skip points where a current speed value was already obtained, allowing the tool to “pick up where it left off”, if it fails. Note that this _expression_ works only for a shapefile, which does not allow null values in a FLOAT or DOUBLE field. When you create a FLOAT or DOUBLE field in a shapefile, all rows get the value 0 by default.

 

I hope that helps. Let me know if you’re still stuck.

 

Best,

 

Jason

 

From: Grant Humphries [mailto:]
Sent: Thursday, June 04, 2015 5:37 PM
To: Jason Roberts
Cc:
Subject: Re: [mget-help] Bug with Hycom data

 

Hey Jason,

Thanks for the response.

Yeah - have tried re-running, restarting computer and ArcGIS, and rebooting router on my side (in case it was a net problem). Kept getting the same error.

Here is the shape file I'm working with (see AllDat.zip)    Also, the output from the script goes:



Executing: HYCOMGLBu0084DInterpolateAtArcGISPoints mag 1day AllDat cur_speed dt # 5 Nearest # # 600 # # # 128 16 16 3 3
Start Time: Thu Jun 04 14:26:25 2015
Running script HYCOMGLBu0084DInterpolateAtArcGISPoints...
Interpolating values for 5477 points of ArcGIS FeatureLayer "AllDat" of ShapeFile "C:\Farallon_Projects\Seabird_fertilization\Seabird_data\SH_by_Year_9KM_grid_effort\AllDat.shp".
Update stopped before all points were processed: 0:00:13 elapsed, 1178 points updated, 0 deleted, 0 unchanged, 0:00:00.011391 per point, 4299 points not processed.
KeyError: u'dn2'
Failed script HYCOMGLBu0084DInterpolateAtArcGISPoints...

Traceback (most recent call last):
  File "C:\Program Files\GeoEco\ArcGISToolbox\Scripts\HYCOMGLBu0084DInterpolateAtArcGISPoints.py", line 5, in <module>
    ExecuteMethodFromCommandLineAsArcGISTool('GeoEco.DataProducts.HYCOM', 'HYCOMGLBu0084D', 'InterpolateAtArcGISPoints')
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\ArcGISScripts.py", line 210, in ExecuteMethodFromCommandLineAsArcGISTool
    exec sourceCode in globals(), locals()
  File "<string>", line 1, in <module>
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataProducts\HYCOM.py", line 3020, in InterpolateAtArcGISPoints
    Interpolator.InterpolateGridsValuesForTableOfPoints(grids, ArcGISTable(points), valueFields, zField=zField, tField=tField, zValue=zValue, where=where, orderBy=orderBy, method=method, noDataValue=noDataValue, useAbsZ=True, seafloorZValue=20000., numBlocksToCacheInMemory=numBlocksToCacheInMemory, xBlockSize=xBlockSize, yBlockSize=yBlockSize, zBlockSize=zBlockSize, tBlockSize=tBlockSize)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\SpatialAnalysis\Interpolation.py", line 568, in InterpolateGridsValuesForTableOfPoints
    value = cls._InterpolatePointOnGrid_NearestNeighbor(grids[i], transformedCoords, noDataValues[i], debugLoggingEnabled)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\SpatialAnalysis\Interpolation.py", line 51, in _InterpolatePointOnGrid_NearestNeighbor
    value = grid.Data.__getitem__(tuple(nearestIndices))
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 3670, in __getitem__
    return getattr(self._Grid(), self._GetMethod)(key)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 3310, in _GetUnscaledDataAsArray
    data, actualNoDataValue = self._ReadNumpyArray(reorderedSliceList)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Virtual.py", line 1744, in _ReadNumpyArray
    self._Cache.insert(0, [regionToRead, self._Grid._ReadNumpyArray(regionToRead)[0]])
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Virtual.py", line 1465, in _ReadNumpyArray
    raise ValueError(_(u'%(dn1)s does not have the same shape as %(dn2)s.') % {u'dn1': grid.DisplayName, u'dn1': self._Grids[0].DisplayName})
KeyError: u'dn2'

Failed to execute (HYCOMGLBu0084DInterpolateAtArcGISPoints).
Failed at Thu Jun 04 14:26:45 2015 (Elapsed Time: 20.14 seconds)


Thanks again for looking into this, and for the quick reply.

Cheers

Grant

 

 

On Thu, Jun 4, 2015 at 1:55 PM Jason Roberts <> wrote:

Grant,

 

I have recently noticed that error:

 

Server error 3: "Invalid Parameter Exception: DArrayDimension.setProjection: Bad Projection Request: stop >= size

occurring interminttantly with the MGET HYCOM tools. I was able to get around it by simply running the tool again. Does that work for you?

 

If not—if you can repro this 100% of the time—could you please send me the complete output of the tool, so I can see your input parameters, and also your shapefile or feature class so that I may repro and debug it?

 

Thanks,

 

Jason

 

From: Grant Humphries [mailto:]
Sent: Thursday, June 04, 2015 4:26 PM
To:
Subject: [mget-help] Bug with Hycom data

 

Hi there,

I'm currently trying to use the Interpolate HYCOM GLBu0.08 4D Variables at Points tool to get HYCOM data for the california current ecosystem.  I'm getting the following error:

ServerError: 'Server error 3: "Invalid Parameter Exception: DArrayDimension.setProjection: Bad Projection Request: stop >= size: 386:386"'
Failed script HYCOMGLBu0084DInterpolateAtArcGISPoints...

Traceback (most recent call last):
  File "C:\Program Files\GeoEco\ArcGISToolbox\Scripts\HYCOMGLBu0084DInterpolateAtArcGISPoints.py", line 5, in <module>
    ExecuteMethodFromCommandLineAsArcGISTool('GeoEco.DataProducts.HYCOM', 'HYCOMGLBu0084D', 'InterpolateAtArcGISPoints')
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\ArcGISScripts.py", line 210, in ExecuteMethodFromCommandLineAsArcGISTool
    exec sourceCode in globals(), locals()
  File "<string>", line 1, in <module>
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataProducts\HYCOM.py", line 3020, in InterpolateAtArcGISPoints
    Interpolator.InterpolateGridsValuesForTableOfPoints(grids, ArcGISTable(points), valueFields, zField=zField, tField=tField, zValue=zValue, where=where, orderBy=orderBy, method=method, noDataValue=noDataValue, useAbsZ=True, seafloorZValue=20000., numBlocksToCacheInMemory=numBlocksToCacheInMemory, xBlockSize=xBlockSize, yBlockSize=yBlockSize, zBlockSize=zBlockSize, tBlockSize=tBlockSize)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\SpatialAnalysis\Interpolation.py", line 568, in InterpolateGridsValuesForTableOfPoints
    value = cls._InterpolatePointOnGrid_NearestNeighbor(grids[i], transformedCoords, noDataValues[i], debugLoggingEnabled)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\SpatialAnalysis\Interpolation.py", line 42, in _InterpolatePointOnGrid_NearestNeighbor
    nearestIndices = grid.GetIndicesForCoords(coords)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 2779, in GetIndicesForCoords
    index = bisect.bisect_right(self.MaxCoords[d], coord)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 3670, in __getitem__
    return getattr(self._Grid(), self._GetMethod)(key)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 2845, in _GetMaxCoords
    return self._GetCoordsForOffset(key, 0.5)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 2881, in _GetCoordsForOffset
    self._CachedTCoords[fixedIncrementOffset] = self._GetTCoordsList(fixedIncrementOffset, self.Shape[coordNum])
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 2625, in _GetShape
    return self.GetLazyPropertyValue('Shape')
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 160, in GetLazyPropertyValue
    value = self._GetLazyPropertyPhysicalValue(name)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Virtual.py", line 1691, in _GetLazyPropertyPhysicalValue
    return self._Grid.GetLazyPropertyValue(name)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 160, in GetLazyPropertyValue
    value = self._GetLazyPropertyPhysicalValue(name)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Virtual.py", line 1446, in _GetLazyPropertyPhysicalValue
    return self._Grids[0].GetLazyPropertyValue(name)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 160, in GetLazyPropertyValue
    value = self._GetLazyPropertyPhysicalValue(name)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataProducts\HYCOM.py", line 2651, in _GetLazyPropertyPhysicalValue
    endDay = grid.CenterCoords['t', -1]
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 3670, in __getitem__
    return getattr(self._Grid(), self._GetMethod)(key)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 2842, in _GetCenterCoords
    return self._GetCoordsForOffset(key, 0.0)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 2857, in _GetCoordsForOffset
    return self._GetCoords(coord, coordNum, slices, sliceDims, fixedIncrementOffset)
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataProducts\HYCOM.py", line 2529, in _GetCoords
    time_ = list(self.ParentCollection._PydapDataset['time'][:])
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\AggregatedModules\pydap\model.py", line 192, in __getitem__
    return self.data[index]
  File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\AggregatedModules\pydap\proxy.py", line 112, in __getitem__
    resp, data = "request(url)   File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\AggregatedModules\pydap\util\http.py", line 57, in request
    raise ServerError(msg)
ServerError: 'Server error 3: "Invalid Parameter Exception: DArrayDimension.setProjection: Bad Projection Request: stop >= size: 386:386"'

Failed to execute (HYCOMGLBu0084DInterpolateAtArcGISPoints).

Projection of my dataset is WGS 1984 - and have been able to use with the other data products (i.e. University of Maine ROMS) ,

Any help on this error would be greatly appreciated

Cheers

Grant Humphries

 

Archives powered by MHonArc.
Top of Page