Chronological Thread 

< Chronological >      < Thread >

---

Hi all and thanks a lot for your quick fit back and all the explanations. I'm not good to understand error message. It's more clear now.

Thanks Jason also, but I've check with my colleagues and they can wait until March for the data I have to extract, so I will follow the news on the web site. I'm taking the opportunity for tree more questions:

-I'm using ArcGIS 10.6 and most of the time the MGET toolbox desapear when I'm closing ArcCatalog

-do you have a mailing list or something to inform people of update or something like that?

- and did you plan to addapt MGET for ArcGISPro ?

Thanks a lot anycas for this wonderfull tools that is MGET

Best regards

Sophie

 

Have you seen the recent announcement (https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=11280). EOSDIS username and password are now required to download most of the Ocean Color data products?   The error you get:

 

RuntimeError: Failed to open netCDF file c:\users\slaran\appdata\local\temp\arc5df1\GeoEco_OceanColorLevel3SMIFileSearcher_Temp_mzv5vr\A2002185.L3m_DAY_POC_poc_4km.nc (a local copy of A2002185.L3m_DAY_POC_poc_4km.nc from NASA OceanColor data archive). The file may not be in netCDF-3 or netCDF-4 format. Detailed error information: netCDF4.Dataset() reported RuntimeError: NetCDF: Unknown file format.

may result from a login failure.  This often produces a “.nc” file which contains the html EODIS login page.    You can check this by replacing “.nc” with “.html” and trying to view the page.

Many automated downloaders (including some of the Python scripts in the OCSSW Processing System) have been failing since the change.   The NASA Ocean Color Forum has several discussions, one for shell scripts using wget or curl and another for downloads of ancillary files needed for local processing, which is done using Python scripts.    For Python scripts that fail, one approach is to generate a list of the URL’s to download and then using wget or curl to download the files from the list.    New versions of the OCSSW Python scripts were released last week.   Wget and curl can store the credentials in a “.netrc” file.  If access to this file is restricted to the user, then it is not necessary to enter usernames or passwords on the command line

or hard code  them in scripts.

 

A couple changes to existing scripts that have been useful:

 

1.       For URL’s of the form https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/ using "ob" in place of "cgi" shortcuts one redirection 2.       Some data sets need the wget option “--auth-no-challenge=on” (see the wget man page) 3.       Some remote sites need to use IPv4, and fail if IPv6 is enabled (I think this is due to some intermediate site not using IPv6. DNS provides the IPv6 address but downloads fail and debugging gives an error to the effect that IPv6 is not available).  Use “wget -4” to disable IPv6.

 

All this a corollary of the rule that security is a pain in the neck (and sometimes even lower).

 

 

 

Hello

I'm using MGET on ArcGis and I recently experiment some troubles to use

Create a Climatological raster for NASA oceanColor Lm SMI

It was runing few weeks ago and now I can't, and receive this kind of message below this e-mail.

Could you let me know if it's come from me/my computer or the tool. It's seems that none of the tools from L3 Product work.

Thanks a lot in advance for your answer and for this very nice Tools, so helpfull!!

Best regards

Sophie Laran

Executing (Create Climatological Rasters for NASA OceanColor L3 SMI Product (3)): OceanColorLevel3SMITimeSeriesCreateClimatologicalArcGISRasters Aqua Daily 4km POC_poc Mean Cumulative C:\Users\slaran\Documents\2_SIG\SPEEcov\POC Replace %(Sensor)s;%(TemporalResolution)s;%(SpatialResolution)s;%(ProductCode)s;%(ClimatologyBinType)s_Climatology;%(Sensor)s_%(ProductCode)s_%(ClimatologyBinName)s_%(Statistic)s.img # 1 1 # "-5 44.5 -0.6 47" 18/01/2019 15/02/2019 60 300 true false
Start Time: Mon Jan 27 09:23:24 2020
Running script OceanColorLevel3SMITimeSeriesCreateClimatologicalArcGISRasters...
Querying cumulative climatological mean of the aqua daily 4km POC_poc from NASA OceanColor, clipped to indices tMin = 6042, tMax = 6070, yMin = 3228, yMax = 3287, xMin = 4200, xMax = 4305.
Query complete: 0:00:00 elapsed, 1 datasets found, 0:00:00.050000 per dataset.
Importing 1 datasets into ArcGIS Folder C:\Users\slaran\Documents\2_SIG\SPEEcov\POC with mode "replace".
Import stopped before all datasets were imported: 0:00:01 elapsed, 0 datasets imported, 0:00:00 per dataset, 1 datasets not imported.
RuntimeError: Failed to open netCDF file c:\users\slaran\appdata\local\temp\arc5df1\GeoEco_OceanColorLevel3SMIFileSearcher_Temp_mzv5vr\A2002185.L3m_DAY_POC_poc_4km.nc (a local copy of A2002185.L3m_DAY_POC_poc_4km.nc from NASA OceanColor data archive). The file may not be in netCDF-3 or netCDF-4 format. Detailed error information: netCDF4.Dataset() reported RuntimeError: NetCDF: Unknown file format.
Failed script OceanColorLevel3SMITimeSeriesCreateClimatologicalArcGISRasters...

Traceback (most recent call last):
  File "C:\Program Files\GeoEco\ArcGISToolbox\Scripts\OceanColorLevel3SMITimeSeriesCreateClimatologicalArcGISRasters.py", line 5, in <module>
    ExecuteMethodFromCommandLineAsArcGISTool('GeoEco.DataProducts.NASA.OceanColor', 'OceanColorLevel3SMITimeSeries', 'CreateClimatologicalArcGISRasters')
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\ArcGISScripts.py", line 210, in ExecuteMethodFromCommandLineAsArcGISTool
    exec sourceCode in globals(), locals()
  File "<string>", line 1, in <module>
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\DataProducts\NASA\OceanColor.py", line 595, in CreateClimatologicalArcGISRasters
    workspace.ImportDatasets(collection.QueryDatasets(), mode, calculateStatistics=calculateStatistics, buildPyramids=buildPyramids)
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line 820, in ImportDatasets
    self._ImportDatasets(datasets, mode.lower(), reportProgress, options)
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\Collections.py", line 695, in _ImportDatasets
    self._ImportDatasetsToPath(pathComponentsForPath[path], datasetsForPath[path], mode, progressReporter, options)
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\ArcGIS.py", line 612, in _ImportDatasetsToPath
    self.DatasetType._ImportDatasetsToPath(os.path.join(self.Path, *pathComponents), sourceDatasets, mode, progressReporter, options)
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\ArcGIS.py", line 1036, in _ImportDatasetsToPath
    elif dataset.DataType not in ArcGISRaster._GDALDataTypeForNumpyDataType.keys():
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line 2673, in _GetDataType
    return self.UnscaledDataType
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line 2697, in _GetUnscaledDataType
    return self.GetLazyPropertyValue('UnscaledDataType')
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line 160, in GetLazyPropertyValue
    value = self._GetLazyPropertyPhysicalValue(name)
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\Virtual.py", line 1970, in _GetLazyPropertyPhysicalValue
    if self._Grids[0].DataType == u'float64':
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line 2671, in _GetDataType
    if self.DataIsScaled:
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line 2692, in _GetDataIsScaled
    return self.GetLazyPropertyValue('ScalingFunction') is not None
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line 160, in GetLazyPropertyValue
    value = self._GetLazyPropertyPhysicalValue(name)
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\Virtual.py", line 563, in _GetLazyPropertyPhysicalValue
    return self._Grid.GetLazyPropertyValue(name)
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line 160, in GetLazyPropertyValue
    value = self._GetLazyPropertyPhysicalValue(name)
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\Virtual.py", line 913, in _GetLazyPropertyPhysicalValue
    return self._Grid.GetLazyPropertyValue(name)
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line 160, in GetLazyPropertyValue
    value = self._GetLazyPropertyPhysicalValue(name)
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\Virtual.py", line 277, in _GetLazyPropertyPhysicalValue
    return self._CachedOldestGrid.GetLazyPropertyValue(name)
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\__init__.py", line 160, in GetLazyPropertyValue
    value = self._GetLazyPropertyPhysicalValue(name)
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\NetCDF.py", line 348, in _GetLazyPropertyPhysicalValue
    v = self._GetVariable()
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\NetCDF.py", line 723, in _GetVariable
    self.ParentCollection._Open()
  File "C:\Python27\ArcGIS10.6\lib\site-packages\GeoEco\Datasets\NetCDF.py", line 288, in _Open
    raise RuntimeError(_(u'Failed to open %(dn)s. The file may not be in netCDF-3 or netCDF-4 format. Detailed error information: netCDF4.Dataset() reported %(e)s: %(msg)s.') % {u'dn': self._DisplayName, u'e': e.__class__.__name__, u'msg': self._Str(e)})
RuntimeError: Failed to open netCDF file c:\users\slaran\appdata\local\temp\arc5df1\GeoEco_OceanColorLevel3SMIFileSearcher_Temp_mzv5vr\A2002185.L3m_DAY_POC_poc_4km.nc (a local copy of A2002185.L3m_DAY_POC_poc_4km.nc from NASA OceanColor data archive). The file may not be in netCDF-3 or netCDF-4 format. Detailed error information: netCDF4.Dataset() reported RuntimeError: NetCDF: Unknown file format.

Failed to execute (Create Climatological Rasters for NASA OceanColor L3 SMI Product (3)).
Failed at Mon Jan 27 09:23:37 2020 (Elapsed Time: 12,89 seconds)

 

-- 

Sophie Laran (PhD)

ADERA

Observatoire PELAGIS - UMS 3462

Université de La Rochelle - CNRS

Pôle Analytique - 5 allées de l'Océan

17 000 LA ROCHELLE - FRANCE

TEL : +33 (0)5 16 49 67 15 (LD)

TEL : +33 (0)5 46 44 99 10 (STD)

http://www.observatoire-pelagis.cnrs.fr

-- 
Sophie Laran (PhD)
ADERA
Observatoire PELAGIS - UMS 3462
Université de La Rochelle - CNRS
Pôle Analytique - 5 allées de l'Océan
17 000 LA ROCHELLE - FRANCE
TEL : +33 (0)5 16 49 67 15 (LD)
TEL : +33 (0)5 46 44 99 10 (STD)
http://www.observatoire-pelagis.cnrs.fr

---

• [mget-help] trouble with Data Product/Nasa Ocean Color/L3, Sophie Laran, 01/27/2020
  • Message not available
    • RE: [mget-help] trouble with Data Product/Nasa Ocean Color/L3, Jason Roberts, 01/27/2020
      • Re: [mget-help] trouble with Data Product/Nasa Ocean Color/L3, Sophie Laran, 01/29/2020

---

Archive powered by MHonArc 2.6.19.