Shapefile to KML using Open Source GIS

[Note: The work described below has resulted in an application named Shape2Earth, which is now available for download.]

One of the things that I have been working on lately (or .. whenever I get the chance) is a Shapefile to KML writer for viewing GIS data in Google Earth that uses open source GIS components. I gained a lot of experience with KML while prototyping an ESRI ArcMap based toolset that used ArcObjects, and thought that an open source equivalent would be worth pursuing.

There were three main reasons that I decided to work on this project. 1) It sounded like a lot of fun, 2) I have been increasingly interested in open source GIS, and 3) the emergence of Google Earth (as well as Google Maps, Yahoo Maps, and Microsoft Live Local) has led to an enormous interest in geospatial visualization outside of the traditional GIS community. There is a massive amount of GIS data available to the public in the form of shapefiles, and a cheap, license free, toolset to get the data into Google Earth might be useful to a lot of people. I also really like the idea of having my own, royalty free, toolset for creating Google Earth based applications.

There are several good open source GIS packages available today, and while none of them come close to matching a professional GIS, quite a few of them are powerful enough to be useful. My personal favorites are OpenJUMP, Quantum GIS (QGIS), and MapWindow GIS. As you might expect, each of these has their strong and weak points. In fact, I have found that if you are willing to jump back and forth between each of them, you can do an enormous amount of serious GIS work.

To develop my Shapefile to KML tool, I decided to use MapWindow GIS. The main reason for choosing this open source GIS was it’s development environment (.NET), the ease with which it was possible to create plugins, and its re-projecting capability (very important if you want to get all of your shapefiles into Google Earths projection system). MapWindow GIS also lets users make shapefiles from scratch pretty quickly, though its editing tools are not as robust as OpenJump or QGIS (which I have often used for editing shapefiles to be converted in MapWindow GIS).

To get an idea of the status of this open source KML initiative (I will certainly have to come up with a better name than that), I thought I would walk through a typical scenario of taking a shapefile off of the internet and placing it in Google Earth.

I’ll start at a web page for accessing GIS data for Austin, Texas (and the regional area) and downloading a shapefile representing School Districts. This shapefile has no projection information, but MapWindow GIS allows us to assign a projection to the shapefile.


In this case, the web site points out that “All data sets are projected into the Texas State Plane Central NAD 83 survey feet coordinate system unless otherwise stated.” So we enter this in the “Choose Projection” form in MapWindow GIS.


We have one more step to go, however, as Google Earth has its own projection System. Now that the shapefile knows what its projection system is, it can convert to WGS84 to be compatible with Google Earth using MapWindow GIS’s “Reproject a Shapefile” tool.


The way that MapWindow GIS handles re-projecting data is to create a new shapefile and write it to the same directory as the shapefile that is being re-projected. The new shapefile has the same name with “_Reprojected” tacked on the end (so, in this case, the school district shapefile I downloaded named “schldist” is re-projected to a new shapefile named “schldist_Reprojected”).

Now that my data is in the proper projection, I can go ahead and add it to Google Earth using MapWindow GIS.


Once the data is in MapWindow GIS, I can make whatever symbolic changes I might want by accessing the layer properties. MapWindow GIS offers the basic coloring schemes that you would find in a typical GIS, such as coloring features by attribute. Once I find a color I am happy with, I can access the Shapefile2KML tool from the menu bar.
At this point in development, I have some pretty basic KML parameter options I can set. The Layer Name is the name that is visible in the table of contents in Google Earth, and the Layer Opacity determines to what degree you can see through the layer. There are currently two ways to determine a height value for each feature. Either a common number can be set that is applied to all features, or a numeric attibute field can be used. The color for each feature is determined by the symbology used in MapWindow GIS.

Once the parameters have been set, I can convert the data to KML. I have a couple of options here as well. I can simply save the KML to the hard drive, or I can save it and have it be automatically loaded into Google Earth. If I want to play around with the data a bit to see how it looks in Google Earth, I can select "Load As Link". This writes and loads a Network Link into Google Earth that references, and therefore, loads the data KML file that was saved on your hard drive. The interesting thing about using this method is that opening a Network Link in Google Earth has a different behaviour than opening a regular KML file. This requires a brief explanation.

If you were to write a function that loads a KML file, and hit it 10 times, it would load that same file 10 time. If you do the same thing with a Network Link, however, it will not load that same file 10 times. Each time you hit the button, the Network Link is reloaded. That means that if you rewrite the Network Link to point to a different KML file before loading it, you would remove the KML data that is currently visible in Google Earth, and replace it with the new KML data.

What this means to us in this case, is that we can keep modifying our shapefile symbology in MapWindow GIS and keep reloading it in Google Earth by selecting the "Reload Link" button. I have found this to be quite handy, and will describe its functionality more later on.


For our demonstration here, I select the "Save and Load" button. After the KML file is written, a dialog box opens to ask me where it should be saved. After I select a location on my hard drive, the KML file is save and then automatically loaded into Google Earth.

This is neat for data that already exists, but I wanted to also take advantage of the creation and editing ability of MapWindow GIS (and other open source GIS systems as well). One way to do this is to find available imagery to digitize off of. I plan to look into using the web service offered by Microsofts TerraServer, but for my initial personal use, I used Google Earth's API to capture their view and write a world file to georeference it for use in GIS.

Currently, I can select "Get Image from GE" from MapWindow GIS (or access a stand-alone version from my desktop) which opens a simple dialog box. I can then zoom into the scene I want in Google Earth and select "Capture View" from the form. If the scenes tilt and orentiation to north are not correct for GIS, I am alerted that the scene needs to change to meet these criteria (which are required for proper alignment in GIS). If I select OK, the scene is automatically moved to zero tilt and zero degrees to north. Selecting "Capture View" again lets me save the view to my hard drive in the form of a jpeg file. Here, I have zoomed into the campus of the University of Northern Iowa and captured my view.

While the resolution of this image is not fantastic, it allows me to do some general sketching using the image as a background reference.

I can load the image into MapWindow GIS, create a brand new shapefile (in this case, a polygon), and start digitizing new polygon features on top of the image. Below, I have traced the outline of one of the buildings on the UNI campus.


I can also load the imagery into other GIS packages for editing. After saving my edited shapefile in MapWindow GIS, I have loaded the image and shapefile into QGIS (below). Once in QGIS, I added another feature and saved my edits.


After I am finished editing my new shapefile, I loaded it back into MapWindow GIS to convert it into KML. Below, I have set the color of the features to silver, set my KML parameters, and then click "Load As Link".


After the data KML is written, I am prompted to save the file to my hard drive. But because I used the "Load As Link" tool, a second file (the Network Link) is also saved. The Network Link KML has the same name as the data KML file with a "_link" added to it. This is the KML file that is actually loaded in Google Earth as soon as I save the file. When the Network Link is loaded into Google Earth, it automatically loads the data KML file that it references.


The interesting thing about this (as described above) is that I can rapidly make modifications to the shapefile and quickly re-load it into Google Earth. The graphics below demonstrate this. I can change the color and KML values, and simply hit "Reload Link" to replace the data into Google Earth



This is just a quick view on the current status of my open source GIS / Google Earth project. Later, I'll describe some additional feature I plan to implement.

Using Google Earth as an Emergency Response Viewer

[NOTE: The process of using Google Earth as an emergency response viewer, described below, can now be done entirely within Google Earth. Refer to Google Earth as a GIS for more details.]

One of the main reasons that I originally started working on an ESRI based extension to translate GIS data into KML for viewing in Google Earth (GE) was to test the viability of using GE as a spatial viewer during an emergency. By the time I was seriously in to my ArcObjects/KML toolset, GE had already been used for Hurricanes Katrina.

During an event (fire, accident, environmental, threat, attack, etc), those in charge need a picture of what is going on in order to make critical decisions. Examples of the kinds of information that they want to see is a cordon around the affected area, the ability to view impacted buildings and other assets, the location of traffic control points, and the location of their first responders (often using Automated Vehicle Location [AVL] technology).

I wanted to build a tool that would work with any ESRI ArcMap based tool. To do this, I built an XML based configuration wizard that is used to define what datasets listed in ArcMap’s table of contents are to be exported to KML on demand. The XML configuration file can also hold symbology information for each of the KML layers. While the ArcObjects extension can use the cartographic information from ArcMap to symbolize features, I have found that colors that work well in ArcMap do not always work as well in GE.

Whenever any data is modified (creation of a cordon, adjustments to a buildings status as people are evacuated), it can be instantly published to KML. The use of a configuration file allows a user to set it up for any ArcMap application. It simply exports all feature classes listed in the configuration file on demand. This allows any emergency response application based on ArcMap to use this tool.

My first test on the system used CH2M HILL’s iCIT emergency management application. As iCIT managed the application, the GE extension would publish the data for viewing in a command center in GE. The KML files were securely sent to the GE client machine using a secure Instant Messenger type of technology. Once the files were update on the client machine, a web service was used to reload the altered file in Google Earth.

iCIT Emergency Management System

GIS Data Dynamically Posted to Google Earth

Google Earth has proven to be an excellent, high level, visualization platform for people who need to make decisions. Most of my current testing has dealt with events that are limited in scope (view samples). I should be able to get my hands on a couple of other ArcMap based command and control application for testing. I would really love to see this type of capability used on a much larger scale to see how it might work.

I also have been working on the plumbing to be able to sit GE on top of a GIS package, and use it to create the datasets that have typically been generated by GIS software (buffers, unions, intersects, selection sets, etc.). I plan to look into using ArcEngine, ArcServer, and/or a combination of open source geospatial processing packages to create much of the functionality currently offered by GIS software using GE itself.

One item of recent interest from the Air Force is the use of the Commercial Joint Mapping Toolkit (C/JMTK) as the geoprocessing power behind GE.

I plan to follow up on this post with some work I have been involved with at the US Air Force Academy with automated vehicle location (AVL) and and unmanned aerial vehicles (UAVs) with streaming video into ArcMap and Google Earth.

Google Earth as My Ubiquitous GIS Interface

I have been working with Geographic Information Systems (GIS) for over 10 years, and have spent the last six working primarily with the DoD (mostly the Air Force). I started off creating geospatial strategic plans for 13 Air Force Base Civil Engineer organizations and a Marine Corps Base in early 2000. That is the year that I also started honing my skills on becoming a more skilled GIS programmer, initially starting with VB 6 and ArcObjects, and then moving on to .NET. In April of 2003, I took a job with CH2M HILL, and started working on-site at the US Air Force Academy in Colorado Springs helping them implement the US Air Force GeoBase Program.

I have had quite a bit of experience with the organizational aspects of GIS, and have struggled to make it relevant to as many people as possible. This has not always been easy. The simple fact is that the great majority of geospatial users in an organization (such as an Air Force installation) need nothing more than a good visualization of their area of interest. This has not always been the strong suite of many GIS packages. I needed something that was simpler and more intuitive than the web-base GIS systems we had typically been deploying.

GIS is used extensively at the Academy. In fact, virtually every cadet there has ESRI’s ArcEditor loaded on their Government issued laptop, and the cadets are required to take introductory GIS courses that become more advanced as they pursue their major.

GIS Day has become a rather large event at the Academy, and is well attended by a number of vendors. One of the vendors that was in attendance on GIS Day 2003 was the Federal Sales representative for Keyhole (now known as Google Earth). The Keyhole demonstration was by far the most popular demonstration for the cadets, as they lined up to plug their address into the Keyhole client and zoom to their homes.

The next year, the same Keyhole representative was in attendance. I had planned to ask more about importing GIS data into the client, but most of the talk centered on Google’s purchase of Keyhole.

This set the stage for 2005. By this time, I had become very interested in open source geospatial capabilities. It started when a colleague of mine asked me to look at JUMP GIS. At that time, I had no idea on the depth of open source capabilities that were out there. The more research I did on the topic of open source GIS, the more excited I became.

This interest coincided with the release of Google Maps, which was followed closely by the free release of the newly christened Google Earth (GE). This, of course, was followed closely by Yahoo Maps and Microsoft’s Virtual Earth. I was, and continue to be, truly amazed by the visualization capabilities that these products provide.

I was also jealous that my ArcIMS and ArcSDE/Oracle system could not operate as smoothly and efficiently as the Google products. Now granted, these are not GIS products, and only have a fraction the power that is available with ArcGIS (I won’t be giving it up anytime soon), but they made me think about my constant wish to have a very simple and pleasing interface for all of the people I wished to serve maps to. If only I could get my data into Google Earth.

In September 2005, while having lunch in downtown Colorado Springs with a number of people from our local CH2M HILL office, I started talking about Google Earth with one of our Enterprise Spatial Solutions Vice Presidents. As our conversation worked its way towards the many ways that Google Earth could help the Air Force mission, he commented “you know that Google Earth has its own language don’t you? It’s called KML, for Keyhole Markup Language.”

I had not known that.

That evening, I downloaded the KML Tutorial off of the Google Earth Forum website.

The next day, I created my first point, line, and polygon in the free version of Google Earth using KML.

The day after that, I started to frantically write ArcObjects code to export the Academy’s building feature class into 3D buildings in Google Earth.

By the end of October, I had a full blown ArcObjects extension with a suite of tools that dynamically convert any ESRI based vector data set to its KML equivalent. I also had started creating Google Earth applications with the Google Earth API using .NET and JavaScript.

One of these applications demonstrates the capability that this technology offers the Air Force, and has been used to brief close to 30 General Officers and the Under-Secretary of the Air Force. I also had (what I consider to be) the unique honor of showing this same application to Google Earth’s General Manager, Strategic Sales Manager, and Chief Engineer at the Google Headquarters in Mountain View, CA in November 2005.

By Christmas, I had started developing plans for creating a license free KML creator. I selected an open source GIS, and wrote an extension to convert shapefiles into KML. I have been very pleased with this experience, and am now interested in creating a complete geospatial visualization system solution using Open Source GIS and Google Earth as the use interface.

As it stands, I believe that I have found the user interface that I have wanted for so long. I also feel that 2005 changed a lot of things in the geospatial world, and I am more than a little bit excited about the way that new and more freely accessible technology is allowing us to view information using our planet as the interface.

One interesting thing that happened during my Google Earth research was that I stumbled on to the concept of BLOGs. There are several very good ones dealing with Google Earth, and I have used them all. I have no intention of trying to replicate them. My primary interest is to talk about methods to make Google Earth more relevant to individuals and business who want to see their own personal data displayed in a way that makes sense to them.

I should have started this BLOG last June. Since I did not, I anticipate that it will take me a while to figure out how to use this effectively. I hope that I’ll eventually publish something here that will be helpful to someone.

Well … here it goes.