Dear Linda,
I’m back and working through my MGET requests...
Does this problem still reproduce for you? I located a machine with ArcGIS 10.3 and installed MGET 0.8a69 on it. I confirmed the machine is running the original
Python 2.7.8 that came with 10.3. I used the same parameters as you did to download the Aqua Monthly 4km PAR_par images. It worked. (I stopped it after two images.)
My theory was going to be that the Python 2.7.8 that came with Arc 10.3 contained a version of the OpenSSL library that was now sufficiently obsolete that NASA’s
server was refusing to talk to it, but that did not seem to be the case, at least for me. We had previously seen this happen with even older versions of ArcGIS, which contained Python versions old enough that their OpenSSL was vulnerable to the “heartbleed”
bug. Some web servers that were patched against heartbleed refused connections from those older versions of Python. At that time we determined that the Python 2.7.8 that came with 10.3 was ok. I suspected that a newer security problem rendered 2.7.8 now problematic,
but apparently not.
So I’m sorry to say that I don’t know what would have caused this problem you’re seeing, and am hoping it was a temporary problem with the server this now resolved...
Jason
From: Linda Waters <>
Sent: Tuesday, July 10, 2018 8:17 AM
To: Jason Roberts <>
Cc:
Subject: Re: [mget-help] help with interpolation of satellite rasters at points (or downloading the rasters themselves)
Update: I spoke too soon. CoRTAD now doesn't give me a ssl error anymore, but it gets stuck at 0% for downloading a raster. I have no idea if that's a related error to the oceancolor problem or something entirely different, but I thought
I'd bring it up since they both had ssl errors before the MGET update, so might be related to some underlying python problem.
On Tue, 10 Jul 2018 at 08:45, Linda Waters <> wrote:
I have ArcGIS 10.3. I just updated to the latest MGET at your suggestion. (I had no idea that I had an old version.) I was using XXX63 and now I've got XXX69. I didn't update the python build because it seemed like it was asking for the
same version I was already using.
I still get an error message, but now it's what I've copied below. The choice of data product and spatial extent doesn't seem to matter but I have obviously not tried all variations. Interestingly, the update did fix a similar error I was
getting when downloading the CoRTAD data products on my most recent attempt.
Executing: OceanColorLevel3SMITimeSeriesCreateArcGISRasters Aqua Monthly 4km PAR_par "C:\Users\Linda\Documents\ArcGIS\ARCGIS data\phase shift paper data and layers\Phase Shift Data Mining\KD490_downloadmonthlycache" Add %(Sensor)s;%(TemporalResolution)s;%(SpatialResolution)s;%(ProductCode)s;%%Y;%(SensorCode).1s%%Y%%j%(EndDate)s.L3m_%(TemporalResolutionCode).3s_%(ProductCode)s_%(SpatialResolution)s.img
# # # # # # 60 300 true false
Start Time: Tue Jul 10 08:31:02 2018
Running script OceanColorLevel3SMITimeSeriesCreateArcGISRasters...
RuntimeError: Failed to query the NASA OceanColor server at
https://oceandata.sci.gsfc.nasa.gov/search/file_search.cgi. The query was retried for 300 seconds without success. Check that the server is operating properly and that your computer can connect to it. If necessary, contact the server's operator for assistance.
If the server and network are operating properly, this problem could be a programming error in this tool. If you suspect one, contact the author of this tool for assistance. Error details: error: [Errno 10054] An existing connection was forcibly closed by
the remote host
Failed script OceanColorLevel3SMITimeSeriesCreateArcGISRasters...
Traceback (most recent call last):
File "C:\Program Files\GeoEco\ArcGISToolbox\Scripts\OceanColorLevel3SMITimeSeriesCreateArcGISRasters.py", line 5, in <module>
ExecuteMethodFromCommandLineAsArcGISTool('GeoEco.DataProducts.NASA.OceanColor', 'OceanColorLevel3SMITimeSeries', 'CreateArcGISRasters')
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\ArcGISScripts.py", line 210, in ExecuteMethodFromCommandLineAsArcGISTool
exec sourceCode in globals(), locals()
File "<string>", line 1, in <module>
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataProducts\NASA\OceanColor.py", line 568, in CreateArcGISRasters
grid = cls(sensor, temporalResolution, spatialResolution, product, timeout, maxRetryTime, cacheDirectory)
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataProducts\NASA\OceanColor.py", line 518, in __init__
super(OceanColorLevel3SMITimeSeries, self).__init__(collection, _expression_=_expression_, reportProgress=False)
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\Virtual.py", line 63, in __init__
self._CachedOldestGrid = collection.GetOldestDataset(_expression_, **options)
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\Datasets\__init__.py", line 758, in GetOldestDataset
dataset = self._GetOldestDataset(parsedExpression, options, parentAttrValues, dateTimeAttrName)
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataProducts\NASA\OceanColor.py", line 157, in _GetOldestDataset
return self._GetOldestOrNewestDataset(parsedExpression, options, False)
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataProducts\NASA\OceanColor.py", line 167, in _GetOldestOrNewestDataset
File "C:\Python27\ArcGIS10.3\lib\site-packages\GeoEco\DataProducts\NASA\OceanColor.py", line 334, in _QueryServer
raise RuntimeError(_(u'Failed to query the NASA OceanColor server at %(url)s. The query was retried for %(retry)i seconds without success. Check that the server is operating properly and that your computer can connect to it. If necessary,
contact the server\'s operator for assistance. If the server and network are operating properly, this problem could be a programming error in this tool. If you suspect one, contact the author of this tool for assistance. Error details: %(e)s: %(msg)s') % {u'url':
url, u'retry': self._MaxRetryTime, u'e': e.__class__.__name__, u'msg': UserPreferredEncodingToUnicode(e)})
RuntimeError: Failed to query the NASA OceanColor server at
https://oceandata.sci.gsfc.nasa.gov/search/file_search.cgi. The query was retried for 300 seconds without success. Check that the server is operating properly and that your computer can connect to it. If necessary, contact the server's operator for assistance.
If the server and network are operating properly, this problem could be a programming error in this tool. If you suspect one, contact the author of this tool for assistance. Error details: error: [Errno 10054] An existing connection was forcibly closed by
the remote host
Failed to execute (OceanColorLevel3SMITimeSeriesCreateArcGISRasters).
Any ideas? I can change the Python code if I have guidance from someone.
On Mon, Jul 9, 2018, 2:57 PM Jason Roberts <> wrote:
Hi Linda,
Sorry for my slow reply. I have been busy with travel and meetings that will finally wrap up at the
end of this week. I will be more available starting next week.
What versions of MGET and ArcGIS do you have? Shaun’s suggestion was the right thing to try (thanks
Shaun!). However I did update MGET to use the https URL instead of http for communicating with the NASA OceanColor. Looks like this change was made January 2017. So perhaps you could try upgrading your MGET? There are some more recent changes that are relevant
to NASA OceanColor data that you might want to have.
However I do not believe that would be the reason for the SSLError that you are now seeing. In the
past, I have seen similar errors when old versions of Python are used to communicate with modern web servers. What version of ArcGIS do you have? (This will determine which version of Python you are using.)
Best,
Jason
From:
<>
On Behalf Of Linda Waters
Sent: Friday, July 6, 2018 4:58 PM
To:
Cc:
Subject: Re: [mget-help] help with interpolation of satellite rasters at points (or downloading the rasters themselves)
Thanks Shaun,
That was a good idea. I edited the web address the https version in the multiple places I could find where it was trying to access the website in the python script. I notice that
I can access the pages through a browser if I use https instead of http, but in MGET now I get a different error message; I think it isn't expecting ssl.
On Fri, 6 Jul 2018 at 16:06, Shaun Walbridge <> wrote:
Hi Linda,
You could try and update the URL, this may be the correct updated location:
https://oceandata.sci.gsfc.nasa.gov/api/file_search
If there isn’t any immediate way to do that through the UI, you can try and modify the underlying Python script that is being executed, OceanColor.py:
https://code.env.duke.edu/projects/mget/browser/MGET/Branches/Jason/PythonPackage/src/GeoEco/DataProducts/NASA/OceanColor.py
That should be located within the installation of MGET. You may want to create a backup copy in case anything goes wrong modifying the file, but it should be easy enough to change
the URL. It’s on line 315, which currently reads:
url = "'https://oceandata.sci.gsfc.nasa.gov/search/file_search.cgi'
Cheers,
Shaun
From:
<> on behalf of Linda Waters <>
Reply-To: Linda Waters <>
Date: Friday, July 6, 2018 at 2:52 PM
To: "" <>
Subject: [mget-help] help with interpolation of satellite rasters at points (or downloading the rasters themselves)
Dear MGET-help/ Jason Roberts:
I am having a problem with accessing some L3 products, either interpolated at a few points for specific dates (preferable) or as rasters. I was trying to access this information
in a hurry to fix a problem with a paper and avoid having to withdraw it and go through the review process all over again, so any quick help you can provide would be very welcome, even if it's to suggest an alternate method. I keep encountering the following
error no matter what L3 product / satellite / time frame I choose:
Running script OceanColorLevel3SMITimeSeriesInterpolateAtArcGISPoints...
Failed to query the NASA OceanColor server at
http://oceandata.sci.gsfc.nasa.gov/search/file_search.cgi. The HTTP request failed with error: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because
connected host has failed to respond. Retrying..
It then retries and ultimately fails, making me think that perhaps the web address for the data has changed.
|