Hi Andrew,

 

Thanks for contacting us about this. Neither of these scenarios is supposed to work in the way you have attempted them, but MGET is relying on the ArcGIS Copy Raster tool to detect the problem and report a sensible error message. In the first scenario, ArcGIS does fail properly but the error message is not useful. In the second case, ArcGIS is not failing it all. I suspect it went ahead and created a 1985 subdirectory inside of the directory that houses the GDB itself. If you view the directory E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb and see a 1985 folder there, that is what happened. It is like it failed to detect that the Winter4kmBSST.gdb directory was actually a file GDB, rather than a regular directory. That is presumably bad, because you're not supposed to tamper with the internal contents of a file GDB directory.

 

You should be able to fix this by modifying the Output Raster Python _expression_ to not include the year as a subdirectory. Try this:

 

os.path.join(outputWorkspace, os.path.basename(inputFile).split(u'.')[0][:13])

 

I removed the middle component of the os.path.join call, which was os.path.dirname(inputFile[len(directoryToSearch)+1:]). That component was obtaining the subdirectory(s) that came between the Directory To Search and the HDF itself. Presumably you have your HDFs sorted into subdirectories: 1985, 1986, and so on.

 

This new _expression_ should copy the rasters directly into the file GDB at the root level. I'm not sure if you're planning to try the entire AVHRR Pathfinder archive. I haven't tried thousands of rasters in a file GDB before, so I can't say how well Arc will stand up to it. ESRI talks up the file GDB as being very robust, so it is definitely worth a try.

 

If you find it inconvenient to have all these rasters in the root level of the file GDB, you can create raster catalogs to hold them. For example, you could create raster catalogs called bsst1985, bsst1986, and so on. Then modify the Python _expression_ to be something like this:

 

os.path.join(outputWorkspace, 'bsst' + os.path.basename(inputFile)[:4], os.path.basename(inputFile).split(u'.')[0][:13])

 

This is extracting the year directly from the file name, rather than assuming you have the rasters stored in subdirectories by year. But you could also have modified the default _expression_ that comes with the tool by just prepending 'bsst' + to the middle component.

 

The MGET tool currently cannot automatically create the raster catalogs for you. But that leads me to enumerate the ways that we could improve the tool for use with GDBs:

 

·         If output raster path includes a raster catalog name, and the raster catalog does not already exist, create the raster catalog:

o    If personal GDB, set Raster Management Type to Unmanaged, otherwise use Managed

o    Consider making this a parameter of the tool

·         Detect problems and report sensible error messages, rather than relying on ArcGIS.

o    Only one subdirectory level (the raster catalog name) is allowed in the output raster path

o    Raster catalog name, if present, must not begin with a number

o    Need to look up other raster naming rules for GDBs

 

I will look into making these improvements. If you have any other suggestions, I'd be happy to hear them.

 

Best,

Jason

 

 

 

Hi Jason et al.,

It looks like there is a minor error in the code for the tool "Find HDF and convert SDS to ArcGIS rasters" - a very nice tool, but if you specify a file GDB as output workspace and use the traverse folders option, it appears the tool will try to create a folder in the the File GDB for each folder traversed which immediately causes it to error when it tries to perform the copyRaster.  If it was a feature dataset you tried to add with the tool (e.g., 1985 below) you can't start with a number, you would have to add bsst1985 for example. The featureclass name can't start with a number either.  The raster name you can fix I see by changing this output, but the folder or dataset issue you can not. Can you add a check to the folder name and have it add a letter before the number? I'm not even sure you can add a feature dataset in this way.  When I made a change to correct the raster output name with a letter first, the tool finished, but there was no ouput!  The error must not have been properly recorded or something.  I'm not even sure what happened.  I include the original tool output for the failed attempt, then the good attempt.  Let me know if you need more info.

Cheers, Andrew


Failed completely.

Executing: HDFFindAndConvertToArcGISRasters "E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly" "E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb" bsst -180 -90 0.0439453 # 19???[1-2].s04m1* true # # "" "" "" "" false false false false PROJCS['World_Mercator',GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Mercator'],PARAMETER['False_Easting',0.0],PARAMETER['False_Northing',0.0],PARAMETER['Central_Meridian',0.0],PARAMETER['Standard_Parallel_1',0.0],UNIT['Meter',1.0]] # # # # # "-28 16 -27 54" # false os.path.join(outputWorkspace, os.path.dirname(inputFile[len(directoryToSearch)+1:]), os.path.basename(inputFile).split(u'.')[0][:13]) os.path false "E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb"
Start Time: Thu Jun 04 09:52:41 2009
Running script HDFFindAndConvertToArcGISRasters...
Finding files and inserting rows into table "work": directory="E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly", wildcard="19???[1-2].s04m1*", searchTree=True, minSize=None, maxSize=None, minDateCreated=None, maxDateCreated=None, minDateModified=None, maxDateModified=None
Finished inserting rows: 0:00:00 elapsed, 21 rows inserted, 0:00:00.010428 per row.
Updating field "outputRaster" in table "work"...
Update complete: 0:00:00 elapsed, 21 rows updated, 0 deleted, 0 unchanged, 0:00:00.000714 per row.
Querying the table "work" to build lists of inputs and outputs...
Query complete: 0:00:00 elapsed, 21 rows retrieved, 0:00:00 per row.
Processing 21 inputs...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1985\198501.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1985\198501...
ERROR 999999: Error executing function.
Failed to execute (CopyRaster).
ArcGISError: An ArcGIS geoprocessing function failed. This usually results from a problem with your inputs. Please check them and try again. Also review any preceding error messages and the detailed error information that appears at the end of this message. If you suspect the failure is due to a programming mistake in this tool or ArcGIS, please contact the author of this tool for assistance. Detailed error information: ArcGIS Geoprocessor object 0x223D8AB8: Invocation of CopyRaster_Management(*args=('C:\\Temp\\GeoEcoTemp_agilbert\\tmpkz5lwm\\output3', 'E:\\Biophysical Oceanograpic Data\\AVHRR\\Pathfinder\\Version5.0\\Seasonal\\Winter\\Winter4kmBSST.gdb\\1985\\198501')) raised ExecuteError: ERROR 999999: Error executing function.
Failed to execute (CopyRaster).
The following consequences resulted from the original error:
Could not copy ArcGIS raster C:\Temp\GeoEcoTemp_agilbert\tmpkz5lwm\output3 to E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1985\198501
Could not convert SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1985\198501.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1985\198501
<class 'GeoEco.ArcGIS.ArcGISError'>: An ArcGIS geoprocessing function failed. This usually results from a problem with your inputs. Please check them and try again. Also review any preceding error messages and the detailed error information that appears at the end of this message. If you suspect the failure is due to a programming mistake in this tool or ArcGIS, please contact the author of this tool for assistance. Detailed error information: ArcGIS Geoprocessor object 0x223D8AB8: Invocation of CopyRaster_Management(*args=('C:\\Temp\\GeoEcoTemp_agilbert\\tmpkz5lwm\\output3', 'E:\\Biophysical Oceanograpic Data\\AVHRR\\Pathfinder\\Version5.0\\Seasonal\\Winter\\Winter4kmBSST.gdb\\1985\\198501')) raised ExecuteError: ERROR 999999: Error executing function.
Failed to execute (CopyRaster).

Failed to execute (HDFFindAndConvertToArcGISRasters).
End Time: Thu Jun 04 09:53:45 2009 (Elapsed Time: 1 minutes 4 seconds)

Completed the following, sort of. The output was not in the GDB. I suspect an error was not being trapped and passed back, but failing and continuing.

Executing: HDFFindAndConvertToArcGISRasters "E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly" "E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb" bsst -180 -90 0.0439453 # 19???[1-2].s04m1* true # # "" "" "" "" false false false false PROJCS['World_Mercator',GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Mercator'],PARAMETER['False_Easting',0.0],PARAMETER['False_Northing',0.0],PARAMETER['Central_Meridian',0.0],PARAMETER['Standard_Parallel_1',0.0],UNIT['Meter',1.0]] # # # # # "-28 16 -27 54" # false os.path.join(outputWorkspace, os.path.dirname(inputFile[len(directoryToSearch)+1:]), "bsstm1" + os.path.basename(inputFile).split(u'.')[0][:13]) os.path false "E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb"
Start Time: Thu Jun 04 10:28:25 2009
Running script HDFFindAndConvertToArcGISRasters...
Finding files and inserting rows into table "work": directory="E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly", wildcard="19???[1-2].s04m1*", searchTree=True, minSize=None, maxSize=None, minDateCreated=None, maxDateCreated=None, minDateModified=None, maxDateModified=None
Finished inserting rows: 0:00:00 elapsed, 21 rows inserted, 0:00:00.003714 per row.
Updating field "outputRaster" in table "work"...
Update complete: 0:00:00 elapsed, 21 rows updated, 0 deleted, 0 unchanged, 0:00:00.000761 per row.
Querying the table "work" to build lists of inputs and outputs...
Query complete: 0:00:00 elapsed, 21 rows retrieved, 0:00:00 per row.
Processing 21 inputs...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1985\198501.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1985\bsstm1198501...
Progress report: 0:01:13 elapsed, 1 inputs processed, 0:01:13.795515 per input, 20 remaining, estimated completion time: 10:54:21 AM.
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1985\198502.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1985\bsstm1198502...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1985\198511.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1985\bsstm1198511...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1985\198512.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1985\bsstm1198512...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1986\198601.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1986\bsstm1198601...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1986\198602.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1986\bsstm1198602...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1986\198611.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1986\bsstm1198611...
Progress report: 0:06:40 elapsed, 7 inputs processed, 0:00:57.265677 per input, 14 remaining, estimated completion time: 10:48:34 AM.
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1986\198612.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1986\bsstm1198612...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1987\198701.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1987\bsstm1198701...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1987\198702.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1987\bsstm1198702...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1987\198711.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1987\bsstm1198711...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1987\198712.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1987\bsstm1198712...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1988\198801.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1988\bsstm1198801...
Progress report: 0:12:19 elapsed, 13 inputs processed, 0:00:56.897947 per input, 8 remaining, estimated completion time: 10:48:26 AM.
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1988\198802.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1988\bsstm1198802...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1988\198811.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1988\bsstm1198811...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1988\198812.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1988\bsstm1198812...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1989\198901.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1989\bsstm1198901...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1989\198902.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1989\bsstm1198902...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1989\198911.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1989\bsstm1198911...
Progress report: 0:17:40 elapsed, 19 inputs processed, 0:00:55.792331 per input, 2 remaining, estimated completion time: 10:48:03 AM.
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1989\198912.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1989\bsstm1198912...
Converting SDS "bsst" in HDF file E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Monthly\1990\199001.s04m1pfv50-bsst-16b.hdf to ArcGIS raster E:\Biophysical Oceanograpic Data\AVHRR\Pathfinder\Version5.0\Seasonal\Winter\Winter4kmBSST.gdb\1990\bsstm1199001...
Processing complete: 0:19:24 elapsed, 21 inputs processed, 0:00:55.448238 per input.
Completed script HDFFindAndConvertToArcGISRasters...
Executed (HDFFindAndConvertToArcGISRasters) successfully.
End Time: Thu Jun 04 10:47:56 2009 (Elapsed Time: 19 minutes 31 seconds)


Andrew Gilbert
Patuxent Wildlife Research Center
U. S. Geological Survey
196 Whitten Rd.
Augusta, ME 04330

Phone: 207-622-8201 X134
Fax: 207-622-8204