Chronological Thread 

< Chronological >      < Thread >

---

Thanks, George, I can confirm that is the problem. Your post is very helpful.
I was not aware of this change; I have not been keeping up on everything
happening at NASA OBPG (sorry about that).

In order for this to work again, it is probably necessary for me to modify
MGET to accept a NASA Earthdata username and password and plumb it through
MGET’s HTTP stack. MGET does not use curl, at least not directly, but the
Python API it relies on does allow for this sort of thing, so it shouldn’t be
too complicated.

I was currently targeting a new version of MGET for March. If I stuck to that
schedule, MGET’s ability to download NASA OceanColor data would be broken
until then. This may be problematic for people. Please contact me privately
if you are actively using MGET to access OceanColor data. If people are
impacted, I’ll see if I can accelerate the work.

Best,
Jason

From: <> On
Behalf Of White, George
Sent: Monday, January 27, 2020 1:44 PM
To:
Cc: Sophie Laran <>
Subject: RE: [mget-help] trouble with Data Product/Nasa Ocean Color/L3

Have you seen the recent announcement
(https://urldefense.proofpoint.com/v2/url?u=https-3A__oceancolor.gsfc.nasa.gov_forum_oceancolor_topic-5Fshow.pl-3Ftid-3D11280&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=xCrbnFZcjrfVEdKQhMVPxwdYw2imyfElky5AJEHT_VA&m=mebiMVdUO_gEw-htRtDB-Nk21epAAIC1KqHWWcIIprQ&s=NXkKd8nF-08usPCgs3ytOBvFhHQdE_bZHkfbS1H8tGw&e=

<https://urldefense.proofpoint.com/v2/url?u=https-3A__oceancolor.gsfc.nasa.gov_forum_oceancolor_topic-5Fshow.pl-3Ftid-3D11280&d=DwMGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=hs6ERJt7ya1IeKAe7E2qJW3BqcrDNsxmGS9TnWxZtU_J3aRD48yYLgnsrfZXbl72&m=LiCDcJ_vrwAu1TSzuerf2PiRz8OLugFKBEn5JL2hCao&s=YEamgg0ipJ9zxDIUv5fo0yu7gJ3d5BUJ7AMVDmwMZKs&e=>).
EOSDIS username and password are now required to download most of the Ocean
Color data products? The error you get:

RuntimeError: Failed to open netCDF file
c:\users\slaran\appdata\local\temp\arc5df1\GeoEco_OceanColorLevel3SMIFileSearcher_Temp_mzv5vr\A2002185.L3m_DAY_POC_poc_4km.nc
(a local copy of A2002185.L3m_DAY_POC_poc_4km.nc from NASA OceanColor data
archive). The file may not be in netCDF-3 or netCDF-4 format. Detailed error
information: netCDF4.Dataset() reported RuntimeError: NetCDF: Unknown file
format.
may result from a login failure. This often produces a “.nc” file which
contains the html EODIS login page. You can check this by replacing “.nc”
with “.html” and trying to view the page.
Many automated downloaders (including some of the Python scripts in the OCSSW
Processing System) have been failing since the change. The NASA Ocean Color
Forum has several discussions, one for shell scripts using wget or curl and
another for downloads of ancillary files needed for local processing, which
is done using Python scripts. For Python scripts that fail, one approach
is to generate a list of the URL’s to download and then using wget or curl to
download the files from the list. New versions of the OCSSW Python scripts
were released last week. Wget and curl can store the credentials in a
“.netrc” file. If access to this file is restricted to the user, then it is
not necessary to enter usernames or passwords on the command line
or hard code them in scripts.

A couple changes to existing scripts that have been useful:


1. For URL’s of the form
https://urldefense.proofpoint.com/v2/url?u=https-3A__oceandata.sci.gsfc.nasa.gov_cgi_getfile_&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=xCrbnFZcjrfVEdKQhMVPxwdYw2imyfElky5AJEHT_VA&m=mebiMVdUO_gEw-htRtDB-Nk21epAAIC1KqHWWcIIprQ&s=7MsW1PlwwtIK3L0dn1_thfBi9OCtQ6hX9JHgRtqQqyk&e=

<https://urldefense.proofpoint.com/v2/url?u=https-3A__oceandata.sci.gsfc.nasa.gov_cgi_getfile_&d=DwMGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=hs6ERJt7ya1IeKAe7E2qJW3BqcrDNsxmGS9TnWxZtU_J3aRD48yYLgnsrfZXbl72&m=LiCDcJ_vrwAu1TSzuerf2PiRz8OLugFKBEn5JL2hCao&s=1AcEvBkiGzz65jkL-vwK78_D1_zmtbCGrM2GgWFEmMA&e=>
using "ob" in place of "cgi" shortcuts one redirection

2. Some data sets need the wget option “--auth-no-challenge=on” (see
the wget man page)

3. Some remote sites need to use IPv4, and fail if IPv6 is enabled (I
think this is due to some intermediate site not using IPv6. DNS provides the
IPv6 address but downloads fail and debugging gives an error to the effect
that IPv6 is not available). Use “wget -4” to disable IPv6.

All this a corollary of the rule that security is a pain in the neck (and
sometimes even lower).

George N. White III

From:
<>

<<>>
On Behalf Of Sophie Laran
Sent: Monday, January 27, 2020 4:42 AM
To: <>
Subject: [mget-help] trouble with Data Product/Nasa Ocean Color/L3


Hello

I'm using MGET on ArcGis and I recently experiment some troubles to use

Create a Climatological raster for NASA oceanColor Lm SMI

It was runing few weeks ago and now I can't, and receive this kind of message
below this e-mail.

Could you let me know if it's come from me/my computer or the tool. It's
seems that none of the tools from L3 Product work.

Thanks a lot in advance for your answer and for this very nice Tools, so
helpfull!!

Best regards

Sophie Laran

Executing (Create Climatological Rasters for NASA OceanColor L3 SMI Product
(3)): OceanColorLevel3SMITimeSeriesCreateClimatologicalArcGISRasters Aqua
Daily 4km POC_poc Mean Cumulative C:\Users\slaran\Documents\2_SIG\SPEEcov\POC
Replace
%(Sensor)s;%(TemporalResolution)s;%(SpatialResolution)s;%(ProductCode)s;%(ClimatologyBinType)s_Climatology;%(Sensor)s_%(ProductCode)s_%(ClimatologyBinName)s_%(Statistic)s.img
# 1 1 # "-5 44.5 -0.6 47" 18/01/2019 15/02/2019 60 300 true false
Start Time: Mon Jan 27 09:23:24 2020
Running script
OceanColorLevel3SMITimeSeriesCreateClimatologicalArcGISRasters...
Querying cumulative climatological mean of the aqua daily 4km POC_poc from
NASA OceanColor, clipped to indices tMin = 6042, tMax = 6070, yMin = 3228,
yMax = 3287, xMin = 4200, xMax = 4305.
Query complete: 0:00:00 elapsed, 1 datasets found, 0:00:00.050000 per dataset.
Importing 1 datasets into ArcGIS Folder
C:\Users\slaran\Documents\2_SIG\SPEEcov\POC with mode "replace".
Import stopped before all datasets were imported: 0:00:01 elapsed, 0 datasets
imported, 0:00:00 per dataset, 1 datasets not imported.
RuntimeError: Failed to open netCDF file
c:\users\slaran\appdata\local\temp\arc5df1\GeoEco_OceanColorLevel3SMIFileSearcher_Temp_mzv5vr\A2002185.L3m_DAY_POC_poc_4km.nc
(a local copy of A2002185.L3m_DAY_POC_poc_4km.nc from NASA OceanColor data
archive). The file may not be in netCDF-3 or netCDF-4 format. Detailed error
information: netCDF4.Dataset() reported RuntimeError: NetCDF: Unknown file
format.
Failed script
OceanColorLevel3SMITimeSeriesCreateClimatologicalArcGISRasters...

Traceback (most recent call last):
File "C:\Program
Files\GeoEco\ArcGISToolbox\Scripts\OceanColorLevel3SMITimeSeriesCreateClimatologicalArcGISRasters.py",
line 5, in <module>

ExecuteMethodFromCommandLineAsArcGISTool('GeoEco.DataProducts.NASA.OceanColor',
'OceanColorLevel3SMITimeSeries', 'CreateClimatologicalArcGISRasters')
File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\ArcGISScripts.py",
line 210, in ExecuteMethodFromCommandLineAsArcGISTool
exec sourceCode in globals(), locals()
File "<string>", line 1, in <module>
File
"C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\DataProducts\NASA\OceanColor.py",
line 595, in CreateClimatologicalArcGISRasters
workspace.ImportDatasets(collection.QueryDatasets(), mode,
calculateStatistics=calculateStatistics, buildPyramids=buildPyramids)
File
"C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line
820, in ImportDatasets
self._ImportDatasets(datasets, mode.lower(), reportProgress, options)
File
"C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\Collections.py",
line 695, in _ImportDatasets
self._ImportDatasetsToPath(pathComponentsForPath[path],
datasetsForPath[path], mode, progressReporter, options)
File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\ArcGIS.py",
line 612, in _ImportDatasetsToPath
self.DatasetType._ImportDatasetsToPath(os.path.join(self.Path,
*pathComponents), sourceDatasets, mode, progressReporter, options)
File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\ArcGIS.py",
line 1036, in _ImportDatasetsToPath
elif dataset.DataType not in
ArcGISRaster._GDALDataTypeForNumpyDataType.keys():
File
"C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line
2673, in _GetDataType
return self.UnscaledDataType
File
"C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line
2697, in _GetUnscaledDataType
return self.GetLazyPropertyValue('UnscaledDataType')
File
"C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line
160, in GetLazyPropertyValue
value = self._GetLazyPropertyPhysicalValue(name)
File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\Virtual.py",
line 1970, in _GetLazyPropertyPhysicalValue
if self._Grids[0].DataType == u'float64':
File
"C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line
2671, in _GetDataType
if self.DataIsScaled:
File
"C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line
2692, in _GetDataIsScaled
return self.GetLazyPropertyValue('ScalingFunction') is not None
File
"C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line
160, in GetLazyPropertyValue
value = self._GetLazyPropertyPhysicalValue(name)
File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\Virtual.py",
line 563, in _GetLazyPropertyPhysicalValue
return self._Grid.GetLazyPropertyValue(name)
File
"C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line
160, in GetLazyPropertyValue
value = self._GetLazyPropertyPhysicalValue(name)
File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\Virtual.py",
line 913, in _GetLazyPropertyPhysicalValue
return self._Grid.GetLazyPropertyValue(name)
File
"C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line
160, in GetLazyPropertyValue
value = self._GetLazyPropertyPhysicalValue(name)
File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\Virtual.py",
line 277, in _GetLazyPropertyPhysicalValue
return self._CachedOldestGrid.GetLazyPropertyValue(name)
File
"C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line
160, in GetLazyPropertyValue
value = self._GetLazyPropertyPhysicalValue(name)
File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\NetCDF.py",
line 348, in _GetLazyPropertyPhysicalValue
v = self._GetVariable()
File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\NetCDF.py",
line 723, in _GetVariable
self.ParentCollection._Open()
File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\NetCDF.py",
line 288, in _Open
raise RuntimeError(_(u'Failed to open %(dn)s. The file may not be in
netCDF-3 or netCDF-4 format. Detailed error information: netCDF4.Dataset()
reported %(e)s: %(msg)s.') % {u'dn': self._DisplayName, u'e':
e.__class__.__name__, u'msg': self._Str(e)})
RuntimeError: Failed to open netCDF file
c:\users\slaran\appdata\local\temp\arc5df1\GeoEco_OceanColorLevel3SMIFileSearcher_Temp_mzv5vr\A2002185.L3m_DAY_POC_poc_4km.nc
(a local copy of A2002185.L3m_DAY_POC_poc_4km.nc from NASA OceanColor data
archive). The file may not be in netCDF-3 or netCDF-4 format. Detailed error
information: netCDF4.Dataset() reported RuntimeError: NetCDF: Unknown file
format.

Failed to execute (Create Climatological Rasters for NASA OceanColor L3 SMI
Product (3)).
Failed at Mon Jan 27 09:23:37 2020 (Elapsed Time: 12,89 seconds)



--



Sophie Laran (PhD)



ADERA



Observatoire PELAGIS - UMS 3462



Université de La Rochelle - CNRS



Pôle Analytique - 5 allées de l'Océan



17 000 LA ROCHELLE - FRANCE



TEL : +33 (0)5 16 49 67 15 (LD)



TEL : +33 (0)5 46 44 99 10 (STD)



https://urldefense.proofpoint.com/v2/url?u=http-3A__www.observatoire-2Dpelagis.cnrs.fr&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=xCrbnFZcjrfVEdKQhMVPxwdYw2imyfElky5AJEHT_VA&m=mebiMVdUO_gEw-htRtDB-Nk21epAAIC1KqHWWcIIprQ&s=NQSELjNCozeFQv6t9-_NgNY7vzQT-8DNp0G0UZ3G9qg&e=

<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.observatoire-2Dpelagis.cnrs.fr&d=DwMDaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=xCrbnFZcjrfVEdKQhMVPxwdYw2imyfElky5AJEHT_VA&m=Ihi0FeyayewQ1zvP_aqFU9KBRwl36rzRwoOeA9aKdHs&s=HHsKnWtEcZ8TfGiriMg8oJ2h5m_4UFgIePLiW0oU6ak&e=>

<<attachment: winmail.dat>>

---

• [mget-help] trouble with Data Product/Nasa Ocean Color/L3, Sophie Laran, 01/27/2020
  • Message not available
    • RE: [mget-help] trouble with Data Product/Nasa Ocean Color/L3, Jason Roberts, 01/27/2020
      • Re: [mget-help] trouble with Data Product/Nasa Ocean Color/L3, Sophie Laran, 01/29/2020

---

Archive powered by MHonArc 2.6.19.