Skip to Content.

mget-help - RE: [mget-help] Error messages running MGET tools

Please Wait...

Subject: Marine Geospatial Ecology Tools (MGET) help

Text archives


RE: [mget-help] Error messages running MGET tools


Chronological Thread 
    < Chronological >      < Thread >
  • From: Jason Roberts <>
  • To: "George N. White III" <>, "" <>
  • Cc: Raul Valente <>
  • Subject: RE: [mget-help] Error messages running MGET tools
  • Date: Sun, 28 Feb 2021 03:11:20 +0000
  • Accept-language: en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=duke.edu; dmarc=pass action=none header.from=duke.edu; dkim=pass header.d=duke.edu; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ew/FReQ52lS1rjtaItGrKHaUvB32WYyyn1xohLPI2jg=; b=gIm653kWXDk1NwjUnqZV9McnmkWvuXZCRas0xxeqH7qehCPmZ523ssKl1K8/R9WqmsB4R0ejgkQZRm9mD5fPngVn6TbOxmDuXQCrWOanjxkhKtR37RtJDkcmt8gJbK/XG7PgS6DacPd/NJZxhuO4acNAcUxdlGUIzmH2bOfGMWE/1eKWW66rPJ6fm/Pex7H923nGC2+G7+uxky8VshtnVpxdVrowYz4+QdswGFk6/ViQxVzehAUPVEJZ3ronYOXDEugz1ygfaMEMScztuhlM3D0GZwW8e1tfl+6bPgYWvY4fkK6Gdr75BVy6DcVYh+DGaRRSBUzs3Pv2Shi4VH0lJQ==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=I/6bNIEeCBDqBZZtkk084WWQvUTvGm+CO3ZNktpJnke76Ar3TTJI1X5CKCJFUiKPwk1EViChJ4PkaOQYpFeV9rDqsUE7EpDAq7OND9uLjDc6D9Ua+zTpPSm7q5I3J9Z9KFLYQrwyUIyW7Epezejq35FK4RaOesxN6jJkYYGcmzhsurArrlfmPWmvinaqxYLM8ervim9haexeL3EkMugDr7gaRA/qZ9PdmL7IBFwWW0cXd/3emG/rBo0tPICfjhECAf0zojYtJWFuXwme6tri0zxh/vO2xcezYZeaiQFwUg36uLATXcf8aXFOSNyjzzGtfluFqQuTWOxKu0LBphZkfw==
  • Authentication-results: gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=duke.edu;

George, thanks for the update on NASA’s changes. Fortunately, I was aware of their change to require EarthData credentials (although hadn’t kept up with the forum change) and reworked MGET’s OceanColor tool last year to handle it. Unfortunately I ran out of time and never pushed the changes to the tree and released a new build of MGET. I have just done that.

 

Raul, as George mentions, to access NASA OceanColor data, you’ll now need to obtain a user name and password from the NASA Earthdata system. I think the correct place to do that is https://urs.earthdata.nasa.gov/users/new. (Note: Duke’s email system will rewrite that URL to a malware protection service. So it looks long and scary and redirects to something like proofpoint dot com, it is probably ok.)

 

After you register for an Earthdata Login, you may have to fill out another form declaring which datasets you are interested in. I cannot remember the exact details of the process. If this is still required, then I presume you should say you are interested in OceanColor data, which comes from NASA Goddard Space Flight Center (GSFC).

 

You should then uninstall your current version of MGET (shut down all ArcGIS programs first) and install MGET 0.8a74 from the MGET website. You can then try the OceanColor tool again.

 

The reworked MGET tool contains boxes for your Earthdata username and password. The rest of the tool’s parameters are similar to before. The tool knows how to authenticate to the Earthdata server and then access the actual data at GSFC. You may need to use a relative new version of Python, because NASA’s servers require that client connections (e.g. from MGET) use SSL, and have strict requirements about the client’s implementation of SSL that are only handled by recent versions of Python. Most ArcGIS users do not install Python themselves; instead they rely on ArcGIS to do it for them. So in practice, this means that you will need a fairly recent version of ArcGIS (maybe 10.3 or later?). If you have an older version of ArcGIS, it is possible to update your version of Python manually. But first it is worth trying it and contacting me with any additional errors you receive.

 

MGET 0.8a74 also fixes the NASA PODAAC MODIS L3 SST problems you noticed. Those problems are related to PODAAC adopting the 2019 reprocessing of the MODIS SST data, as well as changing the organizational scheme on their THREDDS server.

 

The Okubo-Weiss tool is still broken. I have not had time to rework the tool to use the newer AVISO data that is now on CMEMS. It isn’t so much that the data have moved to CMEMS—I already fixed MGET’s ability to access it there—it is that some geographic aspects of the data changed, which requires a deeper adjustment to the Okubo-Weiss code. Meanwhile, Dudley Chelton and collaborators at OSU developed a new eddy detection algorithm (some time ago) that I like better than Okubo-Weiss. See here. I developed an MGET tool to read this database and convert it to a SpatiaLite database, and then extract points and lines from it. Under MGET Data Products, look under Oregon State University CIOSS. But I have not tried this in a while, and when Dudley shifted production of this database to the AVISO team, I suspect they made changes to the database structure, which likely means the current tool will not work. I am planning to use those data myself in about a month, so will be able to get the tool working again then if it is broken. I recommend this eddy database rather than detecting eddies yourself in AVISO SSH data with Okubo-Weiss.

 

I hope that helps!

 

Jason

 

From: <> On Behalf Of George N. White III
Sent: Saturday, February 27, 2021 4:17 PM
To:
Cc: Raul Valente <>
Subject: Re: [mget-help] Error messages running MGET tools

 

On Sat, 27 Feb 2021 at 15:22, Raul Valente <> wrote:

Dear Jason and MGET team,

 

I am writing to request support in the use of the MGET tools for ArcGIS.

 

I am trying to run the Data Products tools and I am having several problems. It seems that there are some issues in accessing data. I only managed to extract some records of CHL (for the OceanColor SMI Product), but then it retrieved an error and it was not successful in extracting the data for all the points.

 

I am attaching a word document with the errors and print screens. Could you provide some assistance and tell me how to get around these issues?

 

NASA now requires an EarthData login to download OceanColor products, if you haven't already done so, you need to register for an EarthData login.   New reprocessings will use new filename conventions.  Many download workflows have problems with the Single-SignOn (SSO) system NASA uses, so it may be necessary to download the files outside MGET and arrange for MGET to use local files.  The NASA Ocean Color Forum had details, but is currently in the process of being transferred to the EarthData forum (for which your SSO login can be used).  I'm copying the text of a relevant announcements as they have not appeared on the EarthData forum:

 

Thank you for your continued support of the OceanColor Forum, but times they are a changin’. On March 1, 2021, we will transition to the Earthdata Forum (https://forum.earthdata.nasa.gov/) for new posts. While the OceanColor Forum will remain accessible for historical knowledge and reference, users should rely on the Earthdata Forum for latest announcements and active user support.   

To ease the transition, many (but not all) of the posts on the OceanColor Forum will transferred to the Earthdata Forum.  Posts to be transferred are those that were made after this forum transitioned to using Earthdata Login for user accounts.  The good news is that if you already have an Earthdata Login account, you'll have ready access to Earthdata Forum, as part of the transition will include an import the accounts associated with the OceanColor Forum to the Earthdata Forum.

While we recommend you continue to post to the OceanColor Forum until the transition on 01 March 2021, please take time between now and then to explore Earthdata Forum, as it is a bit different in organizational structure.

Regards,
Sean

 

To our OceanColor community,

The Ocean Biology Processing Group (OBPG) and the Ocean Biology Distributed
Active Archive Center (OB.DAAC) would like to inform you of a few changes
coming over the next few weeks to months.  These include:

    1) Updated file naming convention (this announcement)
    2) SST reprocessing (this announcement)
    3) Data subscription reconfirmation (details to be provided in November)
    4) EarthData Login requirement for data downloads (details to be provided in November)

1) Updated File Naming Convention

Our current file naming convention was defined over two decades ago with a
single mission in mind (SeaWiFS).  As we took on responsibility for more
missions, we adapted as best we could to the SeaWiFS convention.  Due to the
constraints of the original convention, not all missions were able to fit the
convention nicely, so inconsistencies arose.  With each new mission, we
struggled with how to fit it into this legacy convention.  We realized it was
time to redefine the convention. This change may be jarring to some, but the
consistency achieved is something we have long needed.

A couple examples of the change:

    old: S2003244181700.L2.OC.nc
    new: SEASTAR_SEAWIFS_GAC.20030901T181700.L2.OC.nc
   
    old: A2002191053500.L2_LAC_SST.nc
    new: AQUA_MODIS.20020710T053506.L2.SST.nc
   
Details about this new naming convention are available at:

    https://oceancolor.gsfc.nasa.gov/docs/filenaming-convention/
   
The new naming convention will be applied to each mission/data set as it is
reprocessed by the Ocean Biology Processing Group.  The first data set to get
the new scheme will be the SST data sets for MODIS (Aqua and Terra) and SNPP
VIIRS.

Speaking of SST reprocessing...

2) SST reprocessing

We will be initiating a reprocessing for MODIS (Aqua and Terra) and SNPP VIIRS
SST beginning on 28 October 2019.  The SNPP VIIRS mission will be reprocessed
first, followed by Aqua MODIS and finally Terra MODIS.

R2016.2

The SNPP VIIRS SST reprocessing is primarily to bring consistency to the full
mission.  In mid-2018, a minor update to the VIIRS SST processing was
introduced in the forward stream.  This reprocessing will apply those changes
to the full mission, as well as introduce the use of the GHRSST CMC (Canadian
Meteorological Center) Global Foundation Sea Surface Temperature product as
the reference SST.

    https://oceancolor.gsfc.nasa.gov/reprocessing/r2016/sst/
   
R2019.0

The SST products for the MODIS sensors on both the Aqua and Terra platforms
will be reprocessed to incorporate a revised cloud classification scheme based
on the theory of Alternating Decision Trees (ADtree) developed by Freund and
Mason 1999 and modified by Pfahringer et. al. 2000.  This methodology has been
in use for the VIIRS-SNPP SST product since R2016.0.

Additionally, a correction for atmospheric dust aerosol contaminated nighttime
data based on the work of Luo et al., (2019) has been implemented and, as with
the VIIRS reprocessing, the use of the CMC reference SST will be included.

    https://oceancolor.gsfc.nasa.gov/reprocessing/r2019/sst/
   
Stay tuned for another message to be sent next month describing the changes to
the data subscription service and explaining the EarthData Login requirement
for data downloads.

Regards,
Sean

 

Thank you in advance.

 

Best regards,

Raul Valente

To unsubscribe from this list, visit: https://lists.nicholas.duke.edu/sympa/ca/initiate_unsubscribe/mget-help/gnwiii%40gmail.com



--

George N. White III


Archive powered by MHonArc 2.6.19.

Top of Page