Andy,

Unfortunately it is likely that I will not be at the ESRI Ocean Summit Nov. 7-8. They really want me to come, but my wife is pregnant and the official due date is Oct 24. Unless the baby comes pretty early, allowing our family to get settled in for a few weeks, I’m not going to be available.

Pat Halpin will be going. Part of his presentation will likely reprise a presentation I gave to ESRI in February regarding the need for deep support for time and depth dimensions on geospatial data within all of ESRI’s core products. ESRI’s appears to be using this event as a major requirements-gathering exercise for the ocean community—they want to hear from us what they need to do in their products to better support our needs—so if you are able to make it and convey your opinions, they would appreciate it. Assuming they are getting serious about supporting our community, it could be an opportune time to influence their priorities, to the benefit of us all.

It is good news that you can replicate that problem every time! I will try it this afternoon and let you know what I find. Hopefully we can nail it and get a fix into MGET.

The question of “what to do about missing SST/chlorophyll/whatever data” is pretty complicated. In brief, the usual strategies include:

·         Use temporally-averaged data (e.g. 8-day or monthly) instead of daily data

o    This smooths out strong gradients, etc., so can be a problem depending on your objective

·         Try a different sensor

o    The problem here is that sensors are not perfectly correlated/calibrated with each other. For some discussion of that, go to here and check out the three articles in 2012 that have the phrase “analysis fields inter-comparisons” in their titles.

·         Use a product that tries to merge data from multiple sensors in an intelligent way

o    For SST, try the GHRSST L4 datasets. These are all cloud-free. This one and this one are global, 1 km resolution, although beware they can be highly smoothed and interpolated (you can’t create information out of nothing). MGET has tools to work directly with these, although the downloading can be slow due to necessary decompressing on the NASA server. (I know of a way to partially speed this up, but have not been able to implement yet.)

o    For chlorophyll, you could try this. No direct support in MGET yet, but you can download HDFs and convert them.

·         Use an ocean model that provides simulated output

o    MGET supports direct access to HYCOM and ROMS-CoSiNE, which both have 10 km resolution or so. Beware that they can differ substantially from reality at fine spatial scale.

Hope that helps,

Jason

Hey Jason, thanks for the quick response. Let me know if you ever make it up this way. Are you thinking of going to the ESRI Ocean Summit Nov. 7-8 in Redlands. Notice just came across my desk.

On to the problem. The good news is that I can replicate it, every time.. That is also the bad news L   Attached is the point file we are using. Location is outer WA coast.  I think the dialog box below give you all the info you need to try and run it on your end. I tried several different sensors and Product Codes.

Give me a call if you have any questions. And I will be real mad if it works for you ;-}

I assume that is it SOP to have to hit a couple of different sensors for the same parameter (SST) to fill in missing data. You can see that in the Terra and Aqua SST columns. Any wisdom on the best order of sats/sensors to do this, and thoughts on the comparability of these different sensors.

Andy

Andy,

Nice to see you in Oakland. Sorry I was only in town for 21 hours. We’ll have to make a more concerted effort to catch up next time…

A couple of people reported a similar problem recently. Can you reliably reproduce it, and does it seem to happen on the same file?

Behind the scenes, the tool downloads 2D OceanColor HDFs, decompresses them from bz2 format, and reads them with the pyhdf library. The error occurs at the bz2-decompression step. Python’s built-in bz2 module seems to complain that the files are corrupt. But whenever someone sends in an error report, I try to repro it on the problem file and it always succeeds.

My working hypothesis is that a transient problem with the NASA server causes files to be incompletely downloaded. They made some changes over the summer that seemed to be coincident with this problem starting up. But since I’ve never reproed it myself, I haven’t been able to investigate.

If you can reliably repro it—even if it doesn’t happen on the same file—I can send some debug code that will help us get to the bottom of the issue. Let me know how you fare. I’d like to get it figure out.

Thanks,

Jason

Hi Jason,

Been using your fantastic GeoEco tools, particularly your automated Interpolate satellite images at points tools, but am running into a problem with Ocean Color products.

Successfully got L3 SST from from Terra and Aqua, but the GSFC ocean color is bombing.  Also no luck with SeaWifs or Terra (period of record 5/4/2011 – 8/11/2011).

And, BTW, good to see you at SCB in Oakland. Sorry to have missed you at SCGIS.

Andy

Andrew D. Weiss

Fish Program GIS Lead

WA Dept. of Fish and Wildlife

600 Capitol Way N   MS: 43150

Olympia, WA 98501-1091

Tel: 360-902-2487

Email:

Executing: OceanColorLevel3SMITimeSeriesInterpolateAtArcGISPoints Aqua 8day 4km CHL_chlor_a Samples_2011_assignments_MAC GSFC_OCeanColor_L3_ChloroA_8D_4k GPS_Date Nearest # # # 60 300 # 256 16 16 3

Start Time: Mon Sep 24 10:26:26 2012

Running script OceanColorLevel3SMITimeSeriesInterpolateAtArcGISPoints...

Interpolating values for 737 points of ArcGIS FeatureLayer "Samples_2011_assignments_MAC" of FeatureClass "C:\data\data\Andy\0Projects\OceanGSI\OceanGSI_2011\OceanGSI_2011.gdb\Samples_2011_assignments_MAC_UTM10".

EOFError: compressed file ended before the logical end-of-stream was detected

The following consequences resulted from the original error:

Could not decompress file c:\users\weissadw\appdata\local\temp\GeoEco_OceanColorLevel3SMIFileSearcher_Temp_3fe5a_\A20111212011128.L3m_8D_CHL_chlor_a_4km.bz2 to directory c:\users\weissadw\appdata\local\temp\GeoEco_OceanColorLevel3SMIFileSearcher_Temp_3fe5a_

Update stopped before all points were processed: 0:02:54 elapsed, 0 points updated, 0 deleted, 0 unchanged, 0:00:00 per point, 737 points not processed.

Update in progress: 0:02:54 elapsed, 0 points updated, 0 deleted, 1 unchanged, 0:02:54.250000 per point, 736 remaining, estimated completion time: unknown; more progress needed.

<type 'exceptions.EOFError'>: compressed file ended before the logical end-of-stream was detected

Failed to execute (OceanColorLevel3SMITimeSeriesInterpolateAtArcGISPoints).

Failed at Mon Sep 24 10:29:24 2012 (Elapsed Time: 2 minutes 58 seconds)