style="margin-top:70px;" Clickstream


Visualization as Animation: FAA Traffic Patterns

[FAA visualization]

I've been looking at a lot of visualizations lately to get an idea of both what's possible and what's effective for looking at large volumes of information. A couple weeks ago I came across Aaron Koblin's site. He generated a series of animated visualizations of air traffic over the US based on FAA flight data. Pictured is an image showing traffic volumes over time. From his site:
The following flight pattern visualizations are the result of experiments leading to the project Celestial Mechanics by Scott Hessels and Gabriel Dunne. FAA data was parsed and plotted using the Processing programming environment. The frames were composited with Adobe After Effects and/or Maya and the final piece was highlighted at SIGGRAPH 2005 in the NVIDIA Immersive Dome Experience.
The visual impact of seeing all those planes in the air (up to 19,000 at one time) is amazing. Beautiful movies at his site. Be sure to check out the Celestial Mechanics videos as well.

TDWI Conference Schwag

The last TDWI conference was as eventful as ever. Apart from the new release announcements by Informatica, First Logic and Siebel Analytics (now Oracle), there was plenty of gossip. Unfortunately most of it didn't involve vendors but analyst firms, and a lot of it can't be repeated in print. Find me at the next conference bar for the good stuff.

The IT industry analyst firms are falling on hard times. Money is scarcer than it used to be. Mergers are eating away at the smaller firms, leaving less variety of opinion about the markets and vendors. Some firms are busy laying off their analysts, which makes you wonder how they intend to keep producing analyst reports. Others are trying to get into more consulting to go with market research and analysis. It amazes me that a company would pay money to an analyst firm that consults, since that firm takes money from the vendors. Conflicts of interest are already bad without consulting being heaped onto the analyst's laps.

Evalubase is one market research firm that I think could be interesting. Their concept is that the IT consumers of technology provide feedback in the form of objective surveys in exchange for access to information about the technology they are using and the market. Great idea, but the trick is getting enough participants to provide worthwhile data. If you have a half-dozen responses then the market information might be skewed heavily by one highly favorable or unfavorable rating. When I was CIO of a data syndicator, we were always struggling with sample validity when data didn't come in. There are a lot of other complications besides sample size that can generate misleading statistics. On the whole, I like this model better than the heavily conflicted business model of many analyst firms today. Check them out and give the surveys a try. Maybe objectivity can still win out over money.

The trade show was well-attended this conference. I always make the rounds to see what's new. There wasn't a lot new in the business intelligence space. I was struck by the number of "me too" competitors out there. In many ways, the BI market reminds me of the spreadsheet wars of a decade ago, with a feature introduced by one vendor, then copied by everyone else. I think this market is ripe for both more consolidation and some innovation. Not seeing much of the latter, so I'll be attending this year's O'Reilly Emerging Technology conference in the hope that I can see the (b)leading-edge new ideas (this year's theme is all about information). From their announcement:
"This year's challenge is not about generating digital data—we have more than enough already. It's time to do something with that data. As organic creatures with fallible and finite perception systems, complex desires, and an ever-decreasing amount of time, how do we visualize the data, filter it, remix it, and access it in ways meaningful to us? The opportunity created by the massive data web is one of social good, of personal benefit, and of business advantage."
This is probably the last Etech I'll go to. After a while, many conferences ossify. The conference chairs and speakers get used to each other, fewer outsiders come in, the club doors close, and often at the same time the event grows in attendance and loses the personal feel that made the first few so much fun. I think Etech is peaking and it may be time to find that next great venue.

Honestly, the most interesting research applicable to the BI industry is coming out of fine arts and design schools, not so much from computer science. I think this is a reflection of the difference between the engineering and creative aspects of computing, as much as what happens with cheap access to tools and data by a computer savvy younger generation.

But this post was about TDWI, not some other conference.

So, after checking out the vendors I did the next important thing, which was to drink at their expense. Vendors: the secret to getting people to stay until the doors close at your hospitality suite and not go to the other vendors is to provide enough chairs and small tables for comfortable mingling and gathering. Evidence: two suites at the same time. After the initial talks, one filled up and the other drained out. Same food, same drinks, same lighting and room sizes, just tables and chairs scattered around in the crowded session. The secret to keeping the ginormous hotel bar tab down for an event like this? Don't have chairs and small tables for comfortable mingling and gathering so people leave after the talk part of the event is over.

The next most important thing after all the work is over is to review the trade show giveaways. For your cow-orkers when you return, the most important thing is not what you learned, but what you brought back. Herewith, my take:
[conference tschotkes]
One heavy canvas conference bag
One golf shirt, inconspicuous logo attached
A calculator (they still make those? wow)
A keychain with attached pen (very useful for someone who loses expensive pens a lot, like yours truly)
A keychain with built-in holder for those breath strips that dissolve on your tongue (for those too lazy to suck, unlike yours truly)
One regulation red (evil) lightsabre
A cool refreshing bottle of sugary caffeinated beverage
Two martini glasses (plastic, really) with light-up bases
Four different colors of silly putty eggs
A spiral bound notebook
Some matchbox-style cars
A metal luggage tag guaranteed to light up the ol' airport machines
A USB-powered LED laptop light
Collapsible binoculars for opera-viewing pleasure
An assortment of cheap to decent pens
A small pile of candy (it was Halloween)
A bunch of glass beads that must have been part of a table decoration, discovered in my bag

Some cow-orkers will be very happy with this quarter's take. Others, not so much. Sadly, I missed both the Business Objects user and IBM Integration conferences. No USB thumb drives or other useful gadgets for me until next year. I'm done traveling until January.

DAMA ETL Presentation Slides are Now Available

The slides and notes for the presentation I gave at the Portland DAMA are now converted to a PDF, zipped up and posted here. It's 2 MB in size, so be prepared if you have a slow connection. This is a subset of the course I normally teach at the Data Warehouse Institute, and includes most of the same information. Share and enjoy. Link to zipped PDF

Also, the list of ETL criteria to choose from for RFPs and the like is available in this earlier post.


Data warehousing, business intelligence, IT strategy and architecture, and occasional interesting bits.

Subscribe to XML feed

Bio / About Me

Check out my book

Clickstream data warehousing book cover Buy clickstream data warehousing from

Search this site or  the web

Site search   Web search
powered by FreeFind
Popular Posts
Primate programming.
Why development in crunch mode doesn't work.
Enterprise data modeling sucks big rocks.
XP Exaggerated.
Ping-pong in the matrix.
Time management for anarchists.
Is Ab Initio worth evaluating?
Job posting: omniscient architect.
Why hiring more sales people won't grow revenues faster.
Some resources for Open Source CMS.

Reading List
The Cruise of the Snark
Blue Latitudes
Everyone in Silico
The Klamath Knot
Swarm Intelligence (Bonabeau)
A three year backlog of F&SF

Listening List
Toots and the Maytals
The Buena Vista Social Club
American Idiot

Watching List
Winged Migration Quicktime trailer
Ghengis Blues
Howl's Moving Castls
A Bronx Tale

Daily KOS
Due Diligence
Boing Boing
Kevin Kelly (Recomendo)
Not Geniuses
3 Quarks Daily

War in Context
Valmiki's Ramayana
Choose the Blue
Third Nature
Mark Madsen
The Data Warehouse Institute
James Howard Kunstler
Clickstream Data Warehousing
Technorati Profile

04/01/2003 - 05/01/2003 05/01/2003 - 06/01/2003 06/01/2003 - 07/01/2003 07/01/2003 - 08/01/2003 08/01/2003 - 09/01/2003 09/01/2003 - 10/01/2003 10/01/2003 - 11/01/2003 11/01/2003 - 12/01/2003 12/01/2003 - 01/01/2004 05/01/2004 - 06/01/2004 06/01/2004 - 07/01/2004 07/01/2004 - 08/01/2004 08/01/2004 - 09/01/2004 09/01/2004 - 10/01/2004 10/01/2004 - 11/01/2004 11/01/2004 - 12/01/2004 12/01/2004 - 01/01/2005 01/01/2005 - 02/01/2005 02/01/2005 - 03/01/2005 03/01/2005 - 04/01/2005 05/01/2005 - 06/01/2005 06/01/2005 - 07/01/2005 07/01/2005 - 08/01/2005 08/01/2005 - 09/01/2005 09/01/2005 - 10/01/2005 10/01/2005 - 11/01/2005 11/01/2005 - 12/01/2005 12/01/2005 - 01/01/2006 01/01/2006 - 02/01/2006 03/01/2006 - 04/01/2006 05/01/2006 - 06/01/2006 06/01/2006 - 07/01/2006 07/01/2006 - 08/01/2006 08/01/2006 - 09/01/2006 09/01/2006 - 10/01/2006 10/01/2006 - 11/01/2006 01/01/2007 - 02/01/2007 02/01/2007 - 03/01/2007 03/01/2007 - 04/01/2007 04/01/2007 - 05/01/2007 05/01/2007 - 06/01/2007 06/01/2007 - 07/01/2007 07/01/2007 - 08/01/2007 08/01/2007 - 09/01/2007 09/01/2007 - 10/01/2007 10/01/2007 - 11/01/2007 11/01/2007 - 12/01/2007 12/01/2007 - 01/01/2008 01/01/2008 - 02/01/2008 02/01/2008 - 03/01/2008 03/01/2008 - 04/01/2008 08/01/2008 - 09/01/2008 06/01/2009 - 07/01/2009 08/01/2009 - 09/01/2009 10/01/2009 - 11/01/2009 01/01/2010 - 02/01/2010 09/01/2011 - 10/01/2011 04/01/2013 - 05/01/2013

Powered by Blogger.

Creative Commons License
This work is licensed under this Creative Commons License except where indicated.