Clickstream

Getting Data for Mashups

Getting data is still the hardest part, even with great tools like Dapper and Kapow. They aren't perfect, but neither are the web sites they go after (see previous post about Yahoo!). One thing we learned building the main mashup for the competition was that it's helpful to have an ETL tool around. We used page scrapers and RSS mungers and the like, but the only way to get data from Salesforce.com is to write to their API.

That's no fun. Luckily, we had access to Apatar which has prebuilt connectors for Salesforce, so we were able to go in and fetch the data we wanted without having to resort to client-side javascript calling Salesforce APIs. To do that would have meant coding custom widgets in order to address communication between UI controls, ad there was no way to do that in he time we had.

I love open source. Amazing how quickly OSS data integration has been coming to market. Aside from Apatar there's Talend, Kettle (Pentaho Data Integration) and SnapLogic (though they aren't technically an ETL tool). I saw a mention of JitterBit recently but know nothing about them yet.

Labels: apatar, etl, integration, mashup, mashup camp, mashupcamp4, web2.0

Posted by Mark Saturday, July 21, 2007 10:31:00 PM | permalink |

Comments:

Mark,
SnapLogic can be used for ETL quite easily - we recently posted a set of pipelines based on SugarCRM opportunity analysis. (http://blog.snaplogic.org/?p=63)

# posted by  Anonymous : 7:43 AM

 

SnapLogic is one that we didn't use for this, mostly because it would have added to the learning curve. It's on my list to look at in more detail since I don't know the capabilities.

My impression from the demo at Mashup U was that this was geared more to consuming web-based data than straight-up ETL, though I saw that it could take data from a database and product RSS feeds from it.

# posted by  Mark : 10:40 AM

 

Post a Comment