Archives de catégorie : formats

One simple example of using GDAL/OGR together with Talend or how to get rid of scripts

In this post, you can replace any mention to Talend by Spatial Data Integrator as this latter software is an extension of the initial Talend

One goal of some ETL tools like Talend is to avoid writing multiple and complex lines of code and speed the conception and execution of integration processes. All the operations are designed in a workspace. The administrator picks the components he needs from a palette and links them. Behind, Java or Perl lines of code are generated but the casual administrator wouldn’t need to read nor modify these lines. Somewhat, Talend offers a graphical and friendly way for programming.

Spatial Data Integrator is just Talend to which spatial components and functions have been added. The java libraries behind are GeoTools, Java Topology suite and Sextante. Despite the richness of functionalities, you won’t find some components needed for some specific operations. In particular, Talend SDI doesn’t support as many formats as OGR. Therefore, you can’t convert your files between some specific formats. Besides, in Talend SDI, for the conversion case, it requires knowing the structure of your files (called schemas) in advance. It’s a prerequisite that limits massive format-converting.

Happily, Talend is flexible enough to allow the administrator to enrich the application with additional Java libraries and to launch command line tools. Talend will allow you to easily integrate GDAL/OGR operations inside complex processes with only one line of code. Actually, depending on the utility, it’s the single line of code one is supposed to know.

Many people ask on forums how to use GDAL/OGR over a series of files, for instance how to convert a bunch of ESRI files into KML Files. The given solutions require knowing some elements in shell or batch scripting. Depending on the case, the script could become quite big, thus difficult to maintain (even more when considering one sometimes copies-pastes the lines of code without really understanding them).

Let’s look at the solution given by Tim Sutton, a well-known developer in the OSGeo world, to convert a directory of tiffs to ecw. The SHP to KML script would look alike.

#!/bin/bash
mkdir ecw
for FILE in *.tif
do
BASENAME=$(basename $FILE .tif)
OUTFILE=ecw/${BASENAME}.ecw
echo "Processing: ${BASENAME}.tif"
if [ -f $OUTFILE ] #skip if exists
then
echo "Skipping: $OUTFILE"
else
/usr/local/bin/gdal_translate -of ECW -co LARGE_OK=YES $FILE $OUTFILE
fi
done

As you can see, skills in programming are obvious.

Designing the conversion process in Talend is quite easy as it uses only two components. The first one lists the files inside a folder and the second applies the OGR2OGR command over each of them. This case is one of the simplest example of integrating GDAL/OGR command line tools in Talend.

Let’s examine the job:


Let’s look at the tFileList component:

No explanation needed: just looking at the component properties is explicit enough, far more than multiple lines of code.

Now, let’s look at the tSystem properties in which we launch the OGR2OGR command:

The command is the following one:
« ogr2ogr -f « KML » « +((String)globalMap.get(« tFileList_1_CURRENT_FILEPATH »)).replace(« SHP », »KML »)+ » « +((String)globalMap.get(« tFileList_1_CURRENT_FILEPATH »))

  • ((String)globalMap.get(« tFileList_1_CURRENT_FILEPATH »)) is the complete ESRI file path returned by the tFileList component. You access this global variable by typing the Ctrl-Space shortcut.
  • ((String)globalMap.get(« tFileList_1_CURRENT_FILEPATH »)).replace(« SHP », »KML ») is the complete ESRI File Path in which we replace « SHP » by « KML ». The KML file will be generated in the same folder as the ESRI GIS files.
  •  » escapes the  » character.

This job gives you the main principle for using a command line tool like GDAL/OGR together with Talend. You could do the same with gdalwarp, gdal_translate commands. As the method is mainly graphical and intuitive, it’s easier to develop and to maintain than a shell or batch script. This kind of operation could get more complex when being part of a job that includes other Talend SDI components.

The possibility of transmitting the result of a command to other components can lead to powerful processes, complex, yet at the same easy to improve and maintain. We’ll see one example of command outputting in a future post.

Freemind tip: explore and edit your XML Files

More and more, we use XML files to store data.

The XML file format gives structure to data in a hierarchical way. Thanks to its validation rules, the compliance of data can be checked. Most development languages allow parsing them in order retrieve a specific data.

Here are some examples of XML-based Files in the GIS World:

  • KML, aka Keyhole Markup Language has been developed by google.
  • GML, geographical markup language.
  • GeoRSS, the RSS files with embedded location data.
  • SLD, Styled Layer Descriptor, which permits advanced map renderings.
  • getfeatureinfo request responses returned by a WMS server.

You can read XML files in your web browser but it turns out to be limited and very static.
With Freemind, you can add metadata to nodes like web links, notes, which can be useful if you’d like to collaborate on an XML file before releasing a final version of it.
Also, you can highlight some nodes, add some markers (icons for example), things that are helpful when you start learning a specific XML-based format.

Freemind and XMLs


Freemind is used to represent and organize ideas in a hierarchical, dynamic and graphical way. The Freemind files are XML-based files with a .mm extension.

Let’s take the case of an SLD file. Its structure is rich and complex. I’d like to comfortably explore it before editing it. Freemind will ease navigation through it. Here are the steps to import our SLD file in Freemind.
The operation is valid for any XML files, so you can do it for KMLs.

1 -I open my SLD file in a text editor and I copy all the text

2 – Then I simply paste it in FreeMind. Here is the result:


As you can see, my initial linear text has been rendered in a tree-structured way, which is much more readable and attractive. it’s much more comfortable to edit as well.

By default, after pasting your text, all your nodes are unfolded. You’d surely prefer to get all your nodes folded and unfold only the nodes you want.
To fold all the nodes, just pass your cursor over the root node and select in the navigation item > « fold all the nodes ».

Now, you can use FreeMind features:

  • zooming, automatic folding and unfolding.
  • add nodes, edit nodes
  • take notes,
  • add attributes, icons…
  • attach web links to nodes
  • filtering your nodes by their icons or attributes.

To export your mindmap to an XML:
1 – select the node you want, most often the root node, press copy
2 – then paste in a text editing tool. Now you’ve got your xml. Note that the icons and attributes you added in Freemind have no consequences on the content of your XML but beware of the « note » nodes you might have added. These ones will be considered as XML nodes.

FreeMind is a pretty efficient for taking notes.
One goal of Freemind is to implement a mode in which people would collaborate on a common file over the internet.