Talend and Perl

I’ve downloaded V2.1 of the open source Talend ETL tool; lots of new connectors added and the Java SQLite connector no longer requires a JNI adapter. I’ve evaluated Talend in the past mainly concentrating on its Java code generating capability, this time I revisited the original Perl generator. Why? Well I know Perl, it has a very active community and I’ve successfully used it in the past as a data hacking language on Unix boxes, but I’ve always found using it on Windows a PITA.

For Perl to be a runner as a micro-ETL tool, Windows deployment had to be as simple (for the end-user) as possible. So I decided to invest some time looking at the options for deployment on Win32 and more particularly the use of the PAR module to generate Windows executables. I eventually managed to get PAR to work and now I have a way of shipping a Talend generated Perl ETL transformation to a Windows box without worrying about installing a Perl environment.

With the wealth of Perl data manipulation modules and examples out there, and the ability to create new Talend components via tFlexPerl, I’m seriously considering using Talend/Perl as my micro-ETL environment. The one thing I’m missing is a Perl wrapper for Palo, but I’ve been looking at using SWIG to generate one, looks easy enough, particularly as I would only be interested in exposing two or three Palo functions (adding data to cubes, adding elements to dimensions and issuing a database save).

I know, I know, I’ve been here,there and everywhere looking for my ideal technology. Where will I land? Which option will I finally go with? Stay tuned, as my business plan for “the country datasmith” becomes more concrete, the technology required to support the business becomes clearer, I call this “the perfect storm”. More anon.

Advertisements

2 responses to “Talend and Perl

  1. Tom

    Great minds think alike, I looked as the SWIG many months ago when palo was in version 1, but it wasn’t all that simple there were so many headers and includes in the C files. Then I looked at the XS and again came across stumbling blocks.
    With advent of the HTTP api in version 1.5 things become simpler and to that end I started a project to create a perl API for palo. I’ve got a few objects created and some of the basics are in place, I shall be updating the world about the progress of this project on my blog.

    http://hugoworld.wordpress.com/2007/08/04/the-start-of-something-new/

  2. Hugo,

    That’s great to hear, as you said SWIGing the full API would be a lot of effort, that’s why I intend to just wrap one or two functions in a very simple dll and then SWIG that. I’m also looking at the HTTP api as a means of creating a VBA api (again just a subset) removing the need for installing the full .NET based client in locked down environments. A full Perl HTTP API would be fantastic, keep us informed.

    Tom