Efficient Memory Management in ArcGIS for Server (Save more RAM with these tweaks)

ArcGIS for Server has high affinity when it comes to memory. Services will keep devouring more RAM, especially if those services are not configured properly. No matter how strong your servers get, there will always be limited memory and CPU power. Your number of users will keep increasing the more your services get vital.

I have compiled few tweaks that might help you manage your memory more efficiently and perhaps will save you the trouble and money of buying more RAM.

1. Use Process Low Isolation

Although the use Low Isolation is frowned upon, it has a silver-lining of making your services run with much less memory. This will basically squeeze multiple instances into a single process.

To change this parameter, from the ArcGIS Server Manager, click Services. Then edit the service you want to modify the process isolation properties. Activate the Processes tab and change the Isolation Settings to Low Process Isolation. You can also specify how many instances can run in a single process thus saving more memory.

click to enlarge

 

2. Configure the Services with fewer Instances

You can minimize the maximum number of instances in the pooling options for services. This way you will have fewer instances which consume less memory. You can also increase the waiting time for the users. let them wait for free instances instead of handing them new ones and wasting your memory.

click to enlarge

3. Don't start up instances that are rarely used.

Instances, specially image instances, that are rarely used does not require to be initiated from the start. Set the minimum number of instances for those services to zero to save up some more memory.

click to enlarge

If you are interested for more tips, you may find my new book worth reading Administering ArcGIS for Server. 

Different Architectures and hardware configurations for ArcGIS for Server

There are three main components in ArcGIS for Server, a Web Server which receives requests and forwards them, and a GIS Server which processes and executes the requests. And the configuration store which holds details of the Server Site. The three components can be arranged into different architectures. Following are some illustrated architectures:

It's every man for himself.

This architecture explains the different components of ArcGIS for Server, the GIS Server, the main part which does the processing, the Web Server which is an optional part that receives requests and forwards them to the GIS Server, the geodatabase server which holds your data and last server which holds the Server Site configuration.

The client sends a request to consume a GIS service to the Web Server on port 80 (by default, this can be changed of course) the web server forwards it to the Server Site to a GIS Server, the GIS Server will need to fetch some details from the configuration store about the Server Site, the GIS Server processes the request, it might need to connect to the geodatabase to fetch some data.

This model, It's every man for himself, shows how to distribute servers so everyone can perform one task.

Advantages:

Better management as each node can be monitored separately.

Disadvantages:

Expensive, needs a lot of servers + Heavy networking communication.

I can do that

This model is similar to the previous one, the different is that we got rid of the Storage server for the site configuration and placed it with the Geodatabase Server. The geodatabase server can do the job of the storage server.

Advantages:

Saves one server and higher availability as configuration data is located with the geodatabase.

Disadvantages:

- If the Geodatabase failed the entire Server Site will collapse since the configuration store will no longer be available

-Heavy networking.

Cut the middle man

This model removes the Web Server from the picture thus establishes direct connection with the GIS Server over port 6080

Advantages:

Less networking traffic and configuration

Disadvantages:

-Port 6080 should be opened by the firewall, thus introducing security vulnerabilities.

-No control over who can access the server, which was the job of the web server.

Keep her out of this

This model isolates the geodatabase from the Server Site, the GIS Server will take care of the Config Store. Still the Web Server can control the traffic.

Advantages:

-No Server is required for the Config Store

-

Disadvantages:

-If the GIS Server fails the Server Site is no longer accessible.

-Heavy networking activity

Keep them out this

This model is similar to the previous one, except that the Web Server is removed, thus less configuration and servers. 

The Three Musketeers

This model is setup so that no networking is required at all, the geodatabase, config store are located in the GIS Server. This model is good if the geodatabase is read only. The admin can duplicate multiple GIS Servers with Network Load Balancing between them. In case one fails another can take over.

Advantages:

Redundancy, if a server failed another takes over and have all what it needs

No Networking Traffic, fast retrieval of data. (Data Locality)

Disadvantages:

Redundancy again, since if the geodatabase changed you have to update all GIS Servers.

Mother Teresa

This model is interesting, it is similar to the Three Musketeers , except that the Web Server joins the troop. This model adds the benefit of the Web Server configuration and permissions.

We will introduce another topic where we explain multiple GIS Servers in the server site.

GIS*Plus: The Command line for ESRI Geodatabases

Measuring the performance of a relational database is quite simple. You have all these cool tools in your DBMS which you can use to run query and trace the results. However, in a geodatabase, there is large layer of fat which sits between the DBMS and the client (ArcGIS). Performance cannot be measured as easily, there were some tools like mxdperfstat, but it lacks flexibility and easy of use, plus it is no longer supported by the programmer.

So I thought I can build something that can act like a performance measure for your Geodatabase. A command line where you connect to your Geodatabase and run commands against it. Not only you can connect the geodatabase but you can also specify which geodatabase version to connect.

GIS*Plus is a geodatabase command line utility which allows you to measure performance of your geodatabase. It will establish an SDE connection to your database and will allow you to execute the following to commands so far: (I will work on more commands with time)

ping
The ping command will connect to the feature class and get the feature count, it will initialize any objects that is related to this feature class including relation tables and geometric network.

fetch
The fetch command will connect to the feature class and retrieve all features and loop through them. Then the command line will output some useful statistics about query.


When you first run GIS*Plus, you will be prompted to enter the version name, then the connection database form will be displayed to enter the sde parameters. You can connect to any enterprise geodatabase with this.

After you connect successfully, you can use the command to test your geodatabase. In this example I connected to an Address feature class using ping, you can see that it took 1.79 seconds to complete. The ping operation is equivalent to adding the Address feature class to ArcMap for the first time (without the drawing time of course).

I then run the Fetch command which will actually retrieves all features and loop through them. It will report the time it took to connect to the feature class (0) and the time it took to query (0.04) (40 ms). The 6.01 seconds actually has nothing to do with the database, it is the time that the client took to loop through 230948 records.

Download GIS*Plus for ArcGIS 10.1 SP1

Download GIS*Plus for ArcGIS 10.2

GIS*Plus requires ArcGIS for Desktop (ArcView/ArcEditor) to be installed to check out a license.

Enjoy, and keep me posted with your results and suggestions.