Set up a Postgres Instance to work with ArcGIS [Windows]

We discuss how to set up a postgres instance from scratch, configure the instance to work with ArcGIS and then create an enterprise geodatabase from ArcGIS Desktop and finally we connect to the geodatabase.

My new title: Learning ArcGIS Geodatabase Book, Stay tuned!

Last year I have released my first GIS book to the public with the help of Packt, a great British Publisher. It took me around a year to write, and it was a wonderful journey.

Currently, I have been working again with Pact for few months on a new title on ArcGIS Geodatabses. I am tailoring this one specially for students as I have been received some feedback (thanks alot guys) that Adminstering ArcGIS for Server was a bit advanced for GIS students and doesn't include basic features. Learning ArcGIS Geodatabases will be designed for those who want to start using ArcGIS technology or have been using it and want to learn more about geodatabases.

By contract I cannot exactly release the outline to you guys, however I can assure you that the title will include Creating,  Modeling, Optimizing and even programming geodatabases. I will also include a dedicated chapter for enterprise geodatabases and how you can setup yours from scratch. All these are with complete workable examples, supporting files and exercises. It is going to be fun!

Stay tuned! I am hoping to release it before the end of 2014.

Easiest Method to Backup up File Geodatabases

If you are looking to backup your file geodatabase entirely, then this is the simplest approach. Create a windows batch file that copies your geodatabase to your target media, (external hard drive, flash, tape) and add it to the scheduled task at midnight.

Here is the windows batch script, copy it and paste it on a new Notepad window and save it as backupgdb.bat. Replace C:\Data\mygeodatabase.gdb path in the code with your original gdb folder, and replace E:\Backup with your target backup location.

The script will automatically append the current date so you don't have to worry about that.

 


XCOPY "C:\Data\mygeodatabase.gdb" "E:\Backup\mygeodatabase%date:/=%" /D /E /C /R /I /K /Y
pause

Now add the backupgdb.bat to the scheduled task, following are the necessary steps to do so.

•  From the start menu, type taskschd.msc to open up the Task Scheduler
•  Click on Create Basic Task and type in the name of the task BackupGDB, click Next.
• Select Daily, so the task runs on a daily basis. Click Next.
• Select the time you want this task to run, leave it at midnight and click Next
• Select Start a Program then click Next this way we let
• Windows start our Backupgdb.bat program.
• Browse to your Backupgdb.bat file.
• Click Finish and you are done.

You can create multiple batch files to backup to different locations using the same approach I guess. So you might have Backupgdb_Flash.bat, Backupgdb_NetworkDrive.bat etc..

This method might not be efficient if you want to backup a particular dataset in your geodatabase, as it will simply copy your entire geodatabase to a different location. If you have only a single dataset which is being constantly updated while rest of datasets are static, you will end up copying unchanged redundant data everyday. To copy a particular dataset only I recommend using Geodatabase Replication with a python script instead.

GIS*Plus: The Command line for ESRI Geodatabases

Measuring the performance of a relational database is quite simple. You have all these cool tools in your DBMS which you can use to run query and trace the results. However, in a geodatabase, there is large layer of fat which sits between the DBMS and the client (ArcGIS). Performance cannot be measured as easily, there were some tools like mxdperfstat, but it lacks flexibility and easy of use, plus it is no longer supported by the programmer.

So I thought I can build something that can act like a performance measure for your Geodatabase. A command line where you connect to your Geodatabase and run commands against it. Not only you can connect the geodatabase but you can also specify which geodatabase version to connect.

GIS*Plus is a geodatabase command line utility which allows you to measure performance of your geodatabase. It will establish an SDE connection to your database and will allow you to execute the following to commands so far: (I will work on more commands with time)

ping
The ping command will connect to the feature class and get the feature count, it will initialize any objects that is related to this feature class including relation tables and geometric network.

fetch
The fetch command will connect to the feature class and retrieve all features and loop through them. Then the command line will output some useful statistics about query.


When you first run GIS*Plus, you will be prompted to enter the version name, then the connection database form will be displayed to enter the sde parameters. You can connect to any enterprise geodatabase with this.

After you connect successfully, you can use the command to test your geodatabase. In this example I connected to an Address feature class using ping, you can see that it took 1.79 seconds to complete. The ping operation is equivalent to adding the Address feature class to ArcMap for the first time (without the drawing time of course).

I then run the Fetch command which will actually retrieves all features and loop through them. It will report the time it took to connect to the feature class (0) and the time it took to query (0.04) (40 ms). The 6.01 seconds actually has nothing to do with the database, it is the time that the client took to loop through 230948 records.

Download GIS*Plus for ArcGIS 10.1 SP1

Download GIS*Plus for ArcGIS 10.2

GIS*Plus requires ArcGIS for Desktop (ArcView/ArcEditor) to be installed to check out a license.

Enjoy, and keep me posted with your results and suggestions.

The Unorthodox Way to Compress a Replication-Enabled Versioned GeoDatabase to State 0

If you landed here chances you lost hope of all the links and by-the-book ways to compress your geodatabase. You probably have a versioned geodatabase that no matter how many times you compress, no matter how many users you kick out, how many connections you kill; The state is not set to zero.

You have one version SDE.DEFAULT and it is still pointing to state 8549347 or whatever and this state is referencing thousand of edits. So you are stuck with a huge performance issue on your database because of all the queries to the A and D table. Even if you reconcile and post all versions you still can't seem to get to state 0. Usually replication might cause this, so you might have few hidden versions pointing to state 0 . Therefore compress command will get confused when these versions with state 0.


BEFORE YOU DO THIS, BACKUP YOUR DATABASE, DO AN ORACLE DUMP OR A SQL SERVER BACKUP. ALTHOUGH THIS THING WORKED FOR ME THREE TIMES I CAN'T GUARANTEE JACK.


Buckle up, we are about to open the hood of the SDE Technology and loose few bolts in the SDE engine, change a fuse or two, start the engine manually and put the hood back.


1- Fireup your DBMS, connect as SDE user.

2- Open the SDE.VERSIONS

3- You may skip this to 4 if you just want to do the fix, else continue reading if you want to know how this works

Name: The Version Name
Owner: Who created the version
Version_ID: An ID for the version
Status: A status that tells whether this version if private, public, protected or even system
State_ID: To what state this version is pointing, the state
Description: Text, description of the version
Parent_Name: This is a stupid field, bad ERM design, there is a parent_version_id so we can get the name of the parent. Unless they did it like to avoid joining, thus speed up the query response time, then they are smart and I am sorry.
Parent_Owner: again, stupid field, never mind.
Parent_Version_ID: The parent version id there you go.
creation_time: when did this version is created

4- Now you might have different values, but look at the whole structure it should be the same

5- The geodatabase is not compressing because there are two states pointing to zero which makes the compress command thinks everything is rosy. So take your screwdriver lets screw this DB.

6- We will do a manual reconcile to those two hidden versions to point to the same state id as the default (that is reconciling basically) Change the state 0 in those two version to the state_id of the default as follows

7- Save and close

8- Go run the compress command

sdeversion -o compress [-N]  -u sde -p sde_password -i arcsde_service -s server_name
-D database

You should now see your state tree set to zero.

Mxdperfstat 10.1 Download MXD Performance Statistics

If you are here chances that you were like me searching for an ArcGIS 10.1 version of this wonderful tool. Well, there isn't any, but I managed to make it work by copying some 10 dependency dlls. (Version , System and few others, took me awhile to figure them out)

If you don't know what this tool is, this is it's description from ESRI site

MXDPERFSTAT (ArcGIS 10 and 93) can help diagnose typical MXD document performance problems, e.g. 
• Inefficient scale dependency 
• Slow symbology 
• Large features 
• Projection on the fly 
• Potential database tuning 

System Requirements: 
1. Microsoft .Net Framework 
2. ESRI .NET Assembly 10 or 9.3.1 
a. ArcGIS Desktop with .Net support or 
b. ESRI Engine Runtime 
3. ESRI license: 
a. ArcGIS Engine runtime or 
b. ArcGIS Desktop

This tool will work on ArcGIS 10.1, extract and use,

Download here

If you want the 9.3 or 10 version of this tool click here.

ESRI Geodatabase Downgrade Tool

Most of you must have at least few 9.x (9.2, 9.3, 9.3.1 hopefully no 9.1) while your main production ESRI geodatabase is in 10.x you must face it frustrating that your 10 data won't work on 9.x machines.

Here is a simple tool to use to "Downgrade" any geodatabase to a 9.x version.


Open ArcCatalog

Navigate to the tool.

Double click on Downgrade GDB tool.

Select your 10 gdb

Enter the information

The output will be in the same folder

The result will be the same as your selected gdb plus a 9x suffix.


Enjoy.

Fine-Grained Vs Coarse-Grained Architecture

I got a chance to go through Telvent Software Architecture today and I really liked it. If you take a look at both Telvent and ESRI Object Models you will notice the difference.


Here is the Telvent  Object Model  for all their Products, and this is ESRI's.


ESRI ArcObjects are always referred to as "Fine-grained" while Telvent use the term Coarse Grained. The difference?  its just how much you chop of your design to smaller components.

The smaller your components, the finer it gets, you isolate your code, increase usability to the max. However because your components are so small now you will end up with ALOT of couplings. This will make your architecture very dependant on other objects and it will be very difficult to read as you can see in ESRI object model. Moreover, it is almost impossible to show a fine-grained architecture to upper management since it is too detailed.

Telvent's on the other hand does not go and break down their software components very far, instead they have a little bigger components, yes they cannot reuse all sub components but that makes the architecture decoupled and easily readable and can be presented to someone without architecture skills.

17 Free GIS ESRI Courses [Professional Certifications]

Upon completion of each of the following free Online GIS courses, you get a professional certification that will help you pimp your resume. All you need to do is to follow the training, then do an small exam after that, if you pass the exam you get the certificate.

I just got certified using ArcPad in a 3 Hours course. You will be needing an ESRI Global Account

Field GIS ArcPad Certificate

Free GIS ESRI Courses with Certification

Turning Data into Information Using ArcGIS 10 

Duration: 6 modules (18 hours)

Introduction to ArcGIS 9 Geostatistical Analyst

Duration: 1 module (3 hours)

Getting Started with GIS (for ArcGIS 9.2-9.3)

Duration: 3 modules (9 hours)

GIS for Humanitarian Mine Action

Duration: 8 modules (24 hours)

Getting Started with GIS (for ArcGIS 10)

Duration: 3 modules (9 hours) 

Working with ArcPad 7

Duration: 1 module (3 hours) 

Introduction to GIS Data ReViewer

Duration: 1 module (3 hours) 

Introduction to Map Production System-Atlas

Duration: 1 module (3 hours) 

ArcGIS Data Interoperability Basics

Duration: 1 module (3 hours) 

Using ArcMap in ArcGIS Desktop 10

Duration: 1 module (3 hours) 

Using Python in ArcGIS Desktop 10

Duration: 1 module (3 hours)

Editing in ArcGIS Desktop 10

Duration: 1 module (3 hours)

Mobile GIS: Creating Web Maps for Lightweight Mobile Apps

Duration: 1 module (3 hours) 

Mobile GIS: Using the ArcGIS for iOS Application

Duration: 1 module (3 hours)

Hazus-MH Overview and Installation (for ArcGIS 9.3.1/Hazus-MH MR5)

Duration: 1 module (3 hours)   

Basics of Geographic Coordinate Systems (for ArcGIS 10)

Duration: 1 module (3 hours)  

Field GIS: Collecting and Editing Data Using ArcPad 10

Duration: 1 module (3 hours)