Clickstream

Five Minute Data Integration Podcasts

Podcasts we did on defining data integration technologies just went up at SearchDataManagement today. Each describes a particular technology, what it's used for and when its appropriate. We did four of them on ETL, EII, EDR and EAI, with plans to add several more on topics like master data management and data profiling. It's always strange to hear yourself as other people hear you.

Labels: data federation, data integration, EII, etl, podcast, replication

Posted by Mark Friday, August 31, 2007 12:53:00 PM | permalink | (0) comments

Slides for Data Federation/EII Uses and Abuses

I just uploaded the slides for the data federation/Enterprise Information Integration uses and abuses webcast to SlideShare (embedded below). You can also follow this link to the archived webcast if you want to listen to the talk. Minimal text on the pages so that is probably your best option if you want the details.

Labels: data federation, data integration, EII, events, webcast

Posted by Mark Saturday, August 04, 2007 2:20:00 PM | permalink | (18) comments

Watch a Walkthrough of Our Mashup Camp Contest Winner

Here's a link to coverage of the mashup contest winner (me and Renat Khasanshyn - not often I get to see "me" and "winner" in the same sentence) over at ZDnet: Mashup culture shatters crusty, stodgy old approach to business app dev

They did a nice job with the video of David Berlind interviewing me showing a walkthrough of the mashup and a little about what we had to do to build it. I'd comment on how intelligent that person doing the demo looks, but I'm too modest.

One thing that didn't get mentioned is that Apatar - which Renat used to pull a data feed from Salesforce.com - is open source. It's free as "freedom" and free as in "beer". Hard to beat that.

Sadly, no walkthrough of my answer to the Starbucks finders on Programmable Web, the Starbucks Anti-Finder (on the right). Given how many stores there are, I think the real challenge is showing a map of coffeehouses that aren't Starbucks, so I created a Starbucks locater that can't find Starbucks, even if you were to feed the lat/long position to the map. You're welcome, Brian - now you don't need to see corporate coffeehouses. Version 2 will exclude all public corporations from the list, making cafe hunting even easier for the anti-corporate crowd.

And no, there is no such thing as Jose Cuervo cereal.

Labels: apatar, data integration, mashup, mashup camp, mashupcamp4, open source

Posted by Mark Monday, July 23, 2007 11:08:00 PM | permalink | (0) comments

List of Data Integration Webcast Links for This Year

Here are links to replays for all the webcasts I've done during the past year.

The Hybrid Data Warehouse: Extending the BI Environment with Data Federation
Webcast: 3/21/07

Challenges and Techniques When Hand-coding ETL
Webcast: 6/26/07

Extract-Transform-Load (ETL) Market Overview and Directions
Webcast: 6/13/07

Enterprise Information Integration Technologies: Common Uses and Abuses
Webcast: 2/14/07

Enterprise Integration Series: Evaluation Criteria for Selecting ETL Tools
Webcast: 8/2/06

Labels: data federation, data integration, EII, etl, events, webcasts

Posted by Mark Monday, July 09, 2007 11:58:00 AM | permalink | (0) comments

Slides for the ETL Market Update

These are the slides from the ETL market overview webcast I did for TDWI in May. Share and enjoy.

If the small fonts are hard to read you can go to the slideshare site (link on the lower right) where you can see a full-screen version of these presentations.
The replay of the webcast is now available at the TDWI website.

Labels: data integration, etl, market

Posted by Mark Wednesday, June 27, 2007 6:01:00 PM | permalink | (0) comments

Web Integration Talk in Las Vegas

I'm off to Las Vegas for the Portals, Collaboration and Content Conference to give a talk on data integration for the web. Title of the talk is "Web Data Integration: Methods to Extract and Deliver Data for Portals and Web Applications." I'll be doing a run-through of integration architecture and technology choices to get data from where it is to where you want it. I had to cut back on the part that I find most interesting, getting data off web pages. Scraping, scrAPIs, RSS and the rest of the fun web stuff gets about 10-15 minutes towards the end. I'll probably repurpose the unused information into posts. The slides will make their way online some time in the next couple weeks. I can say one thing - they're pretty relative to some of the other presentations.

Labels: BI, data integration, events, portals

Posted by Mark Monday, May 21, 2007 9:03:00 AM | permalink | (0) comments

When to Consolidate and When to Federate

The will be on data consolidation versus virtualization and the fact that we're starting to need both in the data warehouse (hence the "hybrid data warehouse" in the title).

On-demand access to current data or operational details requires some rethinking of infrastructure. If we want to physically consolidate the data then we need to rethink the extract mechanisms, schemas, storage models and even delivery interfaces to client applications.

The assumption is no longer true that data should always be stored in a central warehouse database. We have technologies that open up access to whole new realms of data: spreadsheets, files, web pages, messages, XML, images, standard industry formats. Centralizing this content isn’t required to manage the schema and relationships for these things – often it can’t.

Data federation / EII tools provide an option that lets us centralize most of the data, but provide access to information that can't easily be consolidated. Unlike direct access to the native sources, these tools allow us to mediate access and provide a single point of management and reuse.

The webcast will go into some more detail on methods of consolidation and where federated access makes sense. Wednesday at 9:00 AM, Pacific.

Labels: data federation, data integration, EII, events

Posted by Mark Sunday, March 18, 2007 7:11:00 PM | permalink | (0) comments

Information Integration Slides Are Posted

I just wrapped up the enterprise information integration (EII) webcast for TDWI so here are the slides. The general goal was to explain a little about what modern EII tools are like and scenarios where they are a good fit. EII went through a small hype cycle in 2004-2005 and kind of fell flat, but the technology has matured enough to fill gaps that ETL and EAI tools aren't well suited to. The slides are available as a PDF from Third Nature.

Next month I'll be doing a webcast on a similar topic, hybrid data warehouse architecture and options for providing access to on-demand/"right time" data. This is more focused on physical versus virtual data consolidation and when to use virtual techniques.

Update: The webcast is available in streaming format from TDWI (they require registration to view).

Labels: data integration, EII

Posted by Mark Wednesday, February 14, 2007 11:06:00 AM | permalink | (0) comments

Data Federation and EII are Underutilized Integration Tools

I'm working my way through the current EII and data federation products as part of research I'm doing for on-demand/real-time data warehousing. The new features in these products - ease of use, non-relational sources, ERP connectors, performance / scaling enhancements, multiple input and output protocols - really make them worth a second look. It's underutilized in current environments that have to deal with on-demand or "right time" data access and multi-format data.

This coming Wednesday (February 14) I'll be doing a short webcast for TDWI on EII covering a little bit of how EII and federation tools work and how they can be used both in and outside the BI environment.

In the past I haven't been a big fan of EII for a few good (I think) reasons:

1. EII/federation vendors are wedged between big ETL vendors on one side and big EAI vendors on the other side (long term product viability).
2. One class of products tended to focus on XML-only interchange, and generally sucked for relational data, particularly in the areas of performance, scalability and ease-of-use.
3. The other class of products focused on distributed queries but didn't work well for anything else, and were mostly limited to relational output.
4. The IT usage scenarios were/are not quite mature enough to provide a solid market.
5. Too expensive relative to the value based on use cases.

I now find myself liking the latest releases of the products. The vendors who survived the mini-hype wave circa 2005 have largely dealt with points 2, 3 and 4 but points 1 and 5 are still true.

Long term viability of standalone vendors is still an issue, though I expect the best few may survive. We're more likely to see EII/federation slip into the data integration stacks or platforms of major vendors and squeezing the smaller niche products out. We're seeing some of that happen already.

The better products still cost too much unless you have the perfect use case. The ROI simply isn't there for real-time data delivery when companies already have alternatives that meet "good enough" criteria; technologies like replication, EAI/queuing software, and CDC/ETL combinations. But...

At the same time, I'm seeing more use cases, and more urgency around some of the existing uses. For a time dashboard hype was lifting EII vendors, but the value/cost ratio of the dashboard use case was poor. Many EII vendors are now chasing the SOA market because EII shows more value there than as an adjunct to support dashboards. Try using EAI tools to make distributed multi-format queries some time. It's not fun.

I'll be writing up more as I run products through their paces. Vendors I'll be doing in-depth work with include IBM, Composite, BEA, Business Objects and several others. Look here over the coming months for more detailed technical writeups. I'm going to be tying portals and mashup servers into the mix so this will be fun.

Labels: data federation, data integration, EII

Posted by Mark Sunday, February 11, 2007 10:30:00 PM | permalink | (0) comments

EII as a Filesystem

I've been digging through a lot of technical details on enterprise information integration (EII) products and technologies lately as I prep for some webcasts on the topic. I also spend a fair amount of time on web integration (accessing web-based data, whether in pages, files or APIs) and I turned up Amit Singh's demo of MacFUSE (embedded below).

What I found novel about it is that FUSE and in particular this demo of MacFUse is the exposing of heterogeneous data as files instead of as web services, APIs or database tables. Instead we're talking about making the data (or fragments of data) available to a standard file browser. Not something I would have thought of since this isn't very "enterprisey", a characteristic of most EII tools. The procFS part of the demo below wasn't that interesting because Unix geeks have had this for years, but RSS as a filesystem and Picasa web APIs as a filesystem were both neat to see.

Labels: data integration, EII, FUSE

Posted by Mark Wednesday, January 31, 2007 1:04:00 PM | permalink | (2) comments