Batch Compact ESRI Map Document (MXD) for 10.2

This tool is from 2012, you guys were using it frequently and asked me to compile it for 10.2 so there it goes. I have also removed the annoying lower case option, now your compacted files are saved under the same name _compact.mxd

Do you have lots of Map Documents with huge sizes?

Compact them all with one click with this tool to more than 60% of their actual size.

Download Compact MXD Tool 10.1

Download Compact MXD Tool 10.2

1) Just open an empty ArcMap session, so the tool can use it to check out license.
2) Run CompactMXD.exe
3) Select the path where your Map Documents are located.
4) Click Compact.

Download Script

I have implemented this guy's idea .

Easiest Method to Backup up File Geodatabases

If you are looking to backup your file geodatabase entirely, then this is the simplest approach. Create a windows batch file that copies your geodatabase to your target media, (external hard drive, flash, tape) and add it to the scheduled task at midnight.

Here is the windows batch script, copy it and paste it on a new Notepad window and save it as backupgdb.bat. Replace C:\Data\mygeodatabase.gdb path in the code with your original gdb folder, and replace E:\Backup with your target backup location.

The script will automatically append the current date so you don't have to worry about that.

 


XCOPY "C:\Data\mygeodatabase.gdb" "E:\Backup\mygeodatabase%date:/=%" /D /E /C /R /I /K /Y
pause

Now add the backupgdb.bat to the scheduled task, following are the necessary steps to do so.

•  From the start menu, type taskschd.msc to open up the Task Scheduler
•  Click on Create Basic Task and type in the name of the task BackupGDB, click Next.
• Select Daily, so the task runs on a daily basis. Click Next.
• Select the time you want this task to run, leave it at midnight and click Next
• Select Start a Program then click Next this way we let
• Windows start our Backupgdb.bat program.
• Browse to your Backupgdb.bat file.
• Click Finish and you are done.

You can create multiple batch files to backup to different locations using the same approach I guess. So you might have Backupgdb_Flash.bat, Backupgdb_NetworkDrive.bat etc..

This method might not be efficient if you want to backup a particular dataset in your geodatabase, as it will simply copy your entire geodatabase to a different location. If you have only a single dataset which is being constantly updated while rest of datasets are static, you will end up copying unchanged redundant data everyday. To copy a particular dataset only I recommend using Geodatabase Replication with a python script instead.

Efficient Memory Management in ArcGIS for Server (Save more RAM with these tweaks)

ArcGIS for Server has high affinity when it comes to memory. Services will keep devouring more RAM, especially if those services are not configured properly. No matter how strong your servers get, there will always be limited memory and CPU power. Your number of users will keep increasing the more your services get vital.

I have compiled few tweaks that might help you manage your memory more efficiently and perhaps will save you the trouble and money of buying more RAM.

1. Use Process Low Isolation

Although the use Low Isolation is frowned upon, it has a silver-lining of making your services run with much less memory. This will basically squeeze multiple instances into a single process.

To change this parameter, from the ArcGIS Server Manager, click Services. Then edit the service you want to modify the process isolation properties. Activate the Processes tab and change the Isolation Settings to Low Process Isolation. You can also specify how many instances can run in a single process thus saving more memory.

click to enlarge

 

2. Configure the Services with fewer Instances

You can minimize the maximum number of instances in the pooling options for services. This way you will have fewer instances which consume less memory. You can also increase the waiting time for the users. let them wait for free instances instead of handing them new ones and wasting your memory.

click to enlarge

3. Don't start up instances that are rarely used.

Instances, specially image instances, that are rarely used does not require to be initiated from the start. Set the minimum number of instances for those services to zero to save up some more memory.

click to enlarge

If you are interested for more tips, you may find my new book worth reading Administering ArcGIS for Server. 

Different Architectures and hardware configurations for ArcGIS for Server

There are three main components in ArcGIS for Server, a Web Server which receives requests and forwards them, and a GIS Server which processes and executes the requests. And the configuration store which holds details of the Server Site. The three components can be arranged into different architectures. Following are some illustrated architectures:

It's every man for himself.

This architecture explains the different components of ArcGIS for Server, the GIS Server, the main part which does the processing, the Web Server which is an optional part that receives requests and forwards them to the GIS Server, the geodatabase server which holds your data and last server which holds the Server Site configuration.

The client sends a request to consume a GIS service to the Web Server on port 80 (by default, this can be changed of course) the web server forwards it to the Server Site to a GIS Server, the GIS Server will need to fetch some details from the configuration store about the Server Site, the GIS Server processes the request, it might need to connect to the geodatabase to fetch some data.

This model, It's every man for himself, shows how to distribute servers so everyone can perform one task.

Advantages:

Better management as each node can be monitored separately.

Disadvantages:

Expensive, needs a lot of servers + Heavy networking communication.

I can do that

This model is similar to the previous one, the different is that we got rid of the Storage server for the site configuration and placed it with the Geodatabase Server. The geodatabase server can do the job of the storage server.

Advantages:

Saves one server and higher availability as configuration data is located with the geodatabase.

Disadvantages:

- If the Geodatabase failed the entire Server Site will collapse since the configuration store will no longer be available

-Heavy networking.

Cut the middle man

This model removes the Web Server from the picture thus establishes direct connection with the GIS Server over port 6080

Advantages:

Less networking traffic and configuration

Disadvantages:

-Port 6080 should be opened by the firewall, thus introducing security vulnerabilities.

-No control over who can access the server, which was the job of the web server.

Keep her out of this

This model isolates the geodatabase from the Server Site, the GIS Server will take care of the Config Store. Still the Web Server can control the traffic.

Advantages:

-No Server is required for the Config Store

-

Disadvantages:

-If the GIS Server fails the Server Site is no longer accessible.

-Heavy networking activity

Keep them out this

This model is similar to the previous one, except that the Web Server is removed, thus less configuration and servers. 

The Three Musketeers

This model is setup so that no networking is required at all, the geodatabase, config store are located in the GIS Server. This model is good if the geodatabase is read only. The admin can duplicate multiple GIS Servers with Network Load Balancing between them. In case one fails another can take over.

Advantages:

Redundancy, if a server failed another takes over and have all what it needs

No Networking Traffic, fast retrieval of data. (Data Locality)

Disadvantages:

Redundancy again, since if the geodatabase changed you have to update all GIS Servers.

Mother Teresa

This model is interesting, it is similar to the Three Musketeers , except that the Web Server joins the troop. This model adds the benefit of the Web Server configuration and permissions.

We will introduce another topic where we explain multiple GIS Servers in the server site.

GIS*Plus: The Command line for ESRI Geodatabases

Measuring the performance of a relational database is quite simple. You have all these cool tools in your DBMS which you can use to run query and trace the results. However, in a geodatabase, there is large layer of fat which sits between the DBMS and the client (ArcGIS). Performance cannot be measured as easily, there were some tools like mxdperfstat, but it lacks flexibility and easy of use, plus it is no longer supported by the programmer.

So I thought I can build something that can act like a performance measure for your Geodatabase. A command line where you connect to your Geodatabase and run commands against it. Not only you can connect the geodatabase but you can also specify which geodatabase version to connect.

GIS*Plus is a geodatabase command line utility which allows you to measure performance of your geodatabase. It will establish an SDE connection to your database and will allow you to execute the following to commands so far: (I will work on more commands with time)

ping
The ping command will connect to the feature class and get the feature count, it will initialize any objects that is related to this feature class including relation tables and geometric network.

fetch
The fetch command will connect to the feature class and retrieve all features and loop through them. Then the command line will output some useful statistics about query.


When you first run GIS*Plus, you will be prompted to enter the version name, then the connection database form will be displayed to enter the sde parameters. You can connect to any enterprise geodatabase with this.

After you connect successfully, you can use the command to test your geodatabase. In this example I connected to an Address feature class using ping, you can see that it took 1.79 seconds to complete. The ping operation is equivalent to adding the Address feature class to ArcMap for the first time (without the drawing time of course).

I then run the Fetch command which will actually retrieves all features and loop through them. It will report the time it took to connect to the feature class (0) and the time it took to query (0.04) (40 ms). The 6.01 seconds actually has nothing to do with the database, it is the time that the client took to loop through 230948 records.

Download GIS*Plus for ArcGIS 10.1 SP1

Download GIS*Plus for ArcGIS 10.2

GIS*Plus requires ArcGIS for Desktop (ArcView/ArcEditor) to be installed to check out a license.

Enjoy, and keep me posted with your results and suggestions.

ArcGIS 10.1 Performance Issues

Since we installed ArcGIS 10.1, we have been experiencing extreme performance issues. We couldn't prove it just by panning and zooming so I had to create a comparison charts to compare different environments.

​We managed to do several testing on the speed of ArcGIS Desktop layer loading between several geodatabase, compared Different Clients (10, 10.1 SP1) vs Oracle and SQL Server.​​ The hardware is relatively the same 16 GB RAM with 8 Core CPU running on Windows 2008 R2 SP1. That is except for the SQL Server DB which is running on 8 GB Windows 2008 R2 SP1. These have exactly the same configurations.

​The log file below shows only loading the a set data set with a geometric network

Scenario A: From Client ArcMap 10.1 to 10.1 Database (Oracle 11g)  

[  1   ] About to add WaterDistributionDataSet

[  1   ]  finished in 0.1 seconds

[  2   ] About to set the source of WaterDistributionDataSet

[  2   ] About to get the feature class object  WaterDistGeomNetwork_Junctions

[  2   ]  finished in 11 seconds

[  3   ] About to get the feature class object  SystemValve

[  3   ]  finished in 0.1 seconds

[  4   ] About to get the feature class object  AirValve

[  4   ]  finished in 0.1 seconds

[  5   ] About to get the feature class object  ServiceValve

[  5   ]  finished in 0.1 seconds

[  6   ] About to get the feature class object  FlowControlValve

[  6   ]  finished in 0.1 seconds

[  7   ] About to get the feature class object  PressureReliefValve

[  7   ]  finished in 0.1 seconds

[  8   ] About to get the feature class object  Fitting

[  8   ]  finished in 0.1 seconds

[  9   ] About to get the feature class object  Junction

[  9   ]  finished in 0.1 seconds

[  10   ] About to get the feature class object  Hydrant

[  10   ]  finished in 0.1 seconds

[  11   ] About to get the feature class object  Pump

[  11   ]  finished in 0.2 seconds

[  12   ] About to get the feature class object  WServicePoint

[  12   ]  finished in 0.1 seconds

[  13   ] About to get the feature class object  Meter

[  13   ]  finished in 0.1 seconds

[  14   ] About to get the feature class object  Tank

[  14   ]  finished in 0.1 seconds

Scenario B: ​From Client ArcMap 10.1 to 10 Database (Oracle 11g)  

[  1   ] About to add WaterDistributionDataSet

[  1   ]  finished in 0.1 seconds

[  2   ] About to set the source of WaterDistributionDataSet

[  2   ] About to get the feature class object  WaterDistGeomNetwork_Junctions

[  2   ]  finished in 13.4 seconds

[  3   ] About to get the feature class object  SystemValve

[  3   ]  finished in 0.7 seconds

[  4   ] About to get the feature class object  AirValve

[  4   ]  finished in 0.7 seconds

[  5   ] About to get the feature class object  ServiceValve

[  5   ]  finished in 0.7 seconds

[  6   ] About to get the feature class object  FlowControlValve

[  6   ]  finished in 0.7 seconds

[  7   ] About to get the feature class object  PressureReliefValve

[  7   ]  finished in 0.8 seconds

[  8   ] About to get the feature class object  Fitting

[  8   ]  finished in 0.8 seconds

[  9   ] About to get the feature class object  Junction

[  9   ]  finished in 0.7 seconds

[  10   ] About to get the feature class object  Hydrant

[  10   ]  finished in 0.7 seconds

[  11   ] About to get the feature class object  Pump

[  11   ]  finished in 1.5 seconds

[  12   ] About to get the feature class object  WServicePoint

[  12   ]  finished in 0.7 seconds

[  13   ] About to get the feature class object  Meter

[  13   ]  finished in 0.7 seconds

[  14   ] About to get the feature class object  Tank

[  14   ]  finished in 0.7 seconds

Scenario C: From Client ArcMap 10 to 10.1 Database (Oracle 11g) 

[  1   ] About to add WaterDistributionDataSet

[  1   ]  finished in 0 seconds

[  2   ] About to set the source of WaterDistributionDataSet

[  2   ] About to get the feature class object  WaterDistGeomNetwork_Junctions

[  2   ]  finished in 1.9 seconds

[  3   ] About to get the feature class object  SystemValve

[  3   ]  finished in 0.1 seconds

[  4   ] About to get the feature class object  AirValve

[  4   ]  finished in 0.1 seconds

[  5   ] About to get the feature class object  ServiceValve

[  5   ]  finished in 0.1 seconds

[  6   ] About to get the feature class object  FlowControlValve

[  6   ]  finished in 0.1 seconds

[  7   ] About to get the feature class object  PressureReliefValve

[  7   ]  finished in 0.1 seconds

[  8   ] About to get the feature class object  Fitting

[  8   ]  finished in 0.2 seconds

[  9   ] About to get the feature class object  Junction

[  9   ]  finished in 0.1 seconds

[  10   ] About to get the feature class object  Hydrant

[  10   ]  finished in 0.1 seconds

[  11   ] About to get the feature class object  Pump

[  11   ]  finished in 0.3 seconds

[  12   ] About to get the feature class object  WServicePoint

[  12   ]  finished in 0.2 seconds

[  13   ] About to get the feature class object  Meter

[  13   ]  finished in 0.1 seconds

[  14   ] About to get the feature class object  Tank

[  14   ]  finished in 0.1 seconds

 ​ 

Scenario D: From Client ArcMap 10.1 to 10.1 Database (SQL Server 2008 R2) [Virtual Machine]

[  1   ] About to add WaterDistributionDataSet

[  1   ]  finished in 0.1 seconds

[  2   ] About to set the source of WaterDistributionDataSet

[  2   ] About to get the feature class object  WaterDistGeomNetwork_Junctions

[  2   ]  finished in 1.5 seconds

[  3   ] About to get the feature class object  SystemValve

[  3   ]  finished in 0.2 seconds

[  4   ] About to get the feature class object  AirValve

[  4   ]  finished in 0.2 seconds

[  5   ] About to get the feature class object  ServiceValve

[  5   ]  finished in 0.2 seconds

[  6   ] About to get the feature class object  FlowControlValve

[  6   ]  finished in 0.2 seconds

[  7   ] About to get the feature class object  PressureReliefValve

[  7   ]  finished in 0.2 seconds

[  8   ] About to get the feature class object  Fitting

[  8   ]  finished in 0.2 seconds

[  9   ] About to get the feature class object  Junction

[  9   ]  finished in 0.2 seconds

[  10   ] About to get the feature class object  Hydrant

[  10   ]  finished in 0.2 seconds

[  11   ] About to get the feature class object  Pump

[  11   ]  finished in 0.3 seconds

[  12   ] About to get the feature class object  WServicePoint

[  12   ]  finished in 0.2 seconds

[  13   ] About to get the feature class object  Meter

[  13   ]  finished in 0.2 seconds

[  14   ] About to get the feature class object  Tank

[  14   ]  finished in 0.2 seconds

Notice that surprisingly​ ​SQL Server while on a virtual machine scored the best performance more than 80% of the 10.1 configuration. This depends on so many factors and configuration but apparently it proves that there is something fishy with 10.1 and Oracle. I would recommend using SQL Server since it complies much better with ESRI technology.