Dear Jason,

     Sorry to be a bother, but I received the following error when trying to fit a random forest model on a much larger dataset (~40k table entries). Have I reached an insurmountable limit or is there any tweaking I can do behind the scenes?

Best,

Bryce

Executing: RandomForestModelFitToArcGISTableUsingFormula sites_nonsites_west "\\PROMETHEUS\Prometheus\CSRM\Benning\DDM_West\random forest\benning_west_rfmodel" "factor(Value) ~ band1_1 + band2_1 + band3_1 + band4_1 + band5_1 + band6_1 + band7_1 + band8_1 + slope_1 + prom_1 + water_cost" 1000 # randomForest # false # # true false false # # # true true false png 1000 3000 3000 10 white

Start Time: Thu Jun 02 13:48:45 2016

Running script RandomForestModelFitToArcGISTableUsingFormula...

Querying ArcGIS table "sites_nonsites_west".

Finished retrieving points: 0:00:25 elapsed, 35594 points retrieved, 0:00:00.000728 per point.

Fitting the random forest using the randomForest package.

Warning in matrix(rfout$xbestsplit, ncol = ntree) :

  Reached total allocation of 2047Mb: see help(memory.size)

Warning in matrix(rfout$xbestsplit, ncol = ntree) :

  Reached total allocation of 2047Mb: see help(memory.size)

Obtaining additional debugging information. This may cause this tool to hang. If you do not see any additional messages after one minute, halt its execution.

Failed script RandomForestModelFitToArcGISTableUsingFormula...

 

Traceback (most recent call last):

  File "C:\Program Files\GeoEco\ArcGISToolbox\Scripts\RandomForestModelFitToArcGISTableUsingFormula.py", line 5, in <module>

    ExecuteMethodFromCommandLineAsArcGISTool('GeoEco.Statistics.Modeling', 'RandomForestModel', 'FitToArcGISTableUsingFormula')

  File "C:\Python27\ArcGIS10.4\lib\site-packages\GeoEco\ArcGISScripts.py", line 210, in ExecuteMethodFromCommandLineAsArcGISTool

    exec sourceCode in globals(), locals()

  File "<string>", line 1, in <module>

  File "C:\Python27\ArcGIS10.4\lib\site-packages\GeoEco\Statistics\Modeling.py", line 724, in FitToArcGISTableUsingFormula

    plotFileFormat, res, width, height, pointSize, bg))

  File "C:\Python27\ArcGIS10.4\lib\site-packages\GeoEco\R.py", line 1084, in __call__

    value = self._Object(str(statements))

  File "C:\Python27\ArcGIS10.4\lib\site-packages\GeoEco\AssimilatedModules\rpy\rpy.py", line 339, in __call__

    return self.eval(self.parse(text=s))

RPy_RException: Error: cannot allocate vector of size 343.3 Mb

 

 

Failed to execute (RandomForestModelFitToArcGISTableUsingFormula).

Failed at Thu Jun 02 13:54:40 2016 (Elapsed Time: 5 minutes 55 seconds)

 

On Wed, Jun 1, 2016 at 11:14 AM Bryce Davenport <> wrote:

Cheers, thank you!

On Tue, May 31, 2016 at 11:06 AM Jason Roberts <> wrote:

Hi Bryce,

 

Glad to know you got it working. A classic paper on the application of random forests to ecology problems is:

 

Cutler DR, Edwards TC Jr, Beard KH, Cutler A, Hess KT, Gibson J, Lawler JJ (2007) Random forests for classification in ecology. Ecology 88: 2783–2792.

 

Hope that helps,

 

Jason

 

From: Bryce Davenport [mailto:]
Sent: Friday, May 27, 2016 1:49 PM
To: Jason Roberts <>
Cc:
Subject: Re: [mget-help] error when running Predict Random Forest From Rasters

 

Dear Jason,

    Thank you for the quick reply! Converting the files to .grd did the trick. I don't suppose you could point me in the direction of literature about interpreting the results?

 

Best,

 

Bryce

 

On Fri, May 27, 2016 at 11:23 AM Jason Roberts <> wrote:

Bryce,

 

This is an unusual error. CopyRaster_management is the Copy Raster tool that is part of ArcGIS itself. The message “Failed to read a pixel block” suggests the tool is having a problem reading the raster. I’m not sure why that would be the case.

 

Try copying the raster yourself with Copy Raster from within your ArcMap session. Run the Copy Raster tool, select the problem raster, and store it someplace else. For the output, make sure there is no file extension (e.g. .tif). The raster will be stored as an ArcInfo Binary Grid. This duplicates what MGET does. (This is an old ESRI-specific format that historically has been reliable with ArcGIS, which is why the MGET tool uses it internally.) If you can get Copy Raster to fail yourself, then we know that it is not something that is specific to MGET. Let me know what happens. Note that ArcInfo Binary Grid format requires the raster name to be 13 characters or less, and it cannot start with a number, and it can only contain letters, numbers, and the underscore. (There might be some other characters allowed, but I don’t recall which.)

 

Then, here are some workarounds to try:

 

1.    Rather than picking the rasters from the drop-down box in the MGET tool, instead click the folder icon and select them through the file system. Perhaps Arc will be better able to read them when accessing them this way, rather than as layers loaded in ArcMap.

 

2.    Copy the problem raster from .tif format to .img format and then try MGET on the .img instead. Perhaps Arc is having a problem reading the .tif file. (This would be odd, since I presume it can display the .tif in ArcMap). If it fails with .img, try copying it to a raster without an extension—ArcInfo Binary Grid format.

 

3.    Try closing ArcMap and running the MGET tool from a stand-alone ArcCatalog session. Perhaps when the raster is loaded into ArcMap it is preventing geoprocessing tools from reading it. (This would be very unusual and likely a bug in ArcGIS.)

 

4.    Make sure you have sufficient disk space on the C: partition to hold a copy of the raster. Although the error did not complain about disk space, this is something that would cause it to fail and perhaps the error message is incorrect. (I doubt it, though.)

 

Best,

Jason

 

From: [mailto:] On Behalf Of Bryce Davenport
Sent: Friday, May 27, 2016 10:44 AM
To:
Subject: [mget-help] error when running Predict Random Forest From Rasters

 

Dear MGET Help,

       This is the error I get when trying to use the Predict Random Forest From Rasters tool. I've changed the configuration of rasters to see if there was an invalid dataset, but it always hangs on the CopyRaster_management tool. Is this a recognized issue or something peculiar to my install?

 

Best,

 

Bryce

 

--

Bryce Davenport
Senior Archaeologist and GIS Specialist, Cultural Site Research and Management

 

2113 St Paul St.
Baltimore, MD 21218
p:410 244-6320
f: 410 244-6324
c:540 292-5483

www.culturalsite.com

***********************

See titles in the ICAHM Springer Press Archaeological Heritage Management Series at http://www.springer.com/series/10186?detailsPage=titles

To unsubscribe from this list, visit: https://lists.nicholas.duke.edu/sympa/ca/initiate_unsubscribe/mget-help/jason.roberts%40duke.edu