Clickstream

Mythology of Big Data keynote from the O'Reilly Strata conference

2011-09-07T12:28:00.001-07:00

This is the keynote I gave at the last O'Reilly Strata conference on the mythology of big data. It's shortened from the original outline, focusing on the primary myth: the 49er-style finding of gold nuggets to make one rich.

Narwals, Jedi of the Sea

2010-01-11T18:08:00.000-08:00

I had no idea that narwhals invented the shish-kabob until I heard this song.

Using YouTube as Your Web Site

2009-10-01T17:12:00.000-07:00

Boone Oakley is an advertising firm that made their website into a YouTube video. Their domain redirects to this video. After you start playing it, you can navigate via buttons integrated into the video to other areas of the "site", which are videos themselves. With embedding, they found a way to make their web site portable across other people's web site.

It's clever and entertaining, but may be the same kind of frustrating that so many flash-driven sites are. Oh, and they have an excellent page on their vision.

How to Speak Hip

2009-08-17T15:39:00.000-07:00

I came across WFMU's Beware of the Blog post on How to Speak Hip which includes mp3s of all the tracks on the 1959 LP. I had this album at one point but lost it. Blew my wig to collar a digital copy. Icing is that this post includes the booklet that came with the LP plus links to Cab Calloway's hepster dictionary and Vout-o-Reenee.

I still need to replace Lord Buckley. Was searching for some missing Buckley when I stumbled on this treasure trove.

Surreal Sit-4-Less and American Express Dispute

2009-06-04T15:05:00.001-07:00

After the Nth go-round of "it's broken", "it's clearance" with American Express on the dispute with Sit-4-Less regarding their ability to ship parts of a product rather than the whole thing, I decided to try a different approach.

I'm now writing to the various company CEOs directly with the following letter. I'm open to other suggestions, so long as they don't involve violence. At least not prosecutable violence.

Here's the most recent letter, with key information redacted. I've always wanted to publish a redacted document. It makes me feel like a journalist protecting his sources, similar to Bob Woodward during Watergate. In this case, it may be more like Carl Kolchak. I liked The Night Stalker series. Inspired me to be a reporter. That was after I decided astronaut wasn't as easy to achieve as Tang made it appear.

June 3, 2009
American Express
PO Box 981532
El Paso, TX 79998
Dispute #xxxxxxx

To whom it may concern,

I am writing in regard to the request for more information on a dispute I have with Sit-4-Less regarding a charge of $$.

Since AMEX last contacted the vendor in April, I received an abused unmarked box with 4 screws in it (on April 22). The intent of this gift was a mystery until I noticed that the return address is where Sit-4-Less ships from. While the screws would allow me to attach the back of a chair to its base, they don't solve the primary problem area, namely "back of chair".

I need an office chair that can be used for sitting as opposed to the one from Sit-4-Less which is more suitable as a sculpture or stand for a pet parrot.

Not having a parrot, I am at a loss as to what I should do with this "chair". I suspect the assembly and adjustment manual that accompanies a Herman Miller chair explains how to use it for pets other than parrots when one can't use it for sitting. Sadly, this manual was lost along with a few other parts that might contribute to the classical definition of a chair when Sit-4-Less improperly packed it for shipment.

To help explain my position, I've included a diagram showing the consensus definition of a "Herman Miller office chair." Note: consensus is defined by my wife asking "How exactly are you going to sit on that thing? It's missing parts." which demonstrates our agreement. (Apologies for the crudeness of my diagram. I often slept in art class due to the early morning start.)

Diagram 1: purpose of chair parts

Diagram 2: components of a chair

Diagram 3: Einsteinian thought experiment involving chair, me, and a parrot

Missing any of these elements changes an office chair into another class of product, say "hurricane-strength paperweight" or "perch for large avian beasts."

While Sit-4-Less demonstrates true American business efficiency by selling partial chairs as clearance items, it would be helpful if they would label them as such. Perhaps they could call them something other than "chair" or add a footnote that reads "item can't be used for sitting, but will be helpful to parrot owners."

I suspect the "4-Less" part of Sit-4-Less derives from their ability to ship one chair's parts to two customers, a wholesale savings to them of 50%, not to mention lower shipping costs. I'm partly to blame for assuming the name was an indication that I would benefit from the "4-Less" part as well as the "Sit".

If these practices are what AMEX expects from businesses, I will plan better in the future by exploring the types of pets that can use half-assembled products as perches. A vulture would be cool. I've always been fond of iguanas too. Do you think they would get along?

Regardless, this complaint has been open for almost a year for one reason: Sit-4-Less sent me an improperly packed product in a damaged box and made no effort to correct the problem for almost 9 months, during which time I bought a new (real) chair. My niece would like me to add that the new chair's wheels do, in fact, go round and round.

I would appreciate your attention the matter of getting me a refund or return. I can be reached at ###-###-####, day or night, although at night I sometimes stub my toe on the Sit-4-Less bird perch while trying to reach the phone, so please don't take the swearing personally as that is not my intent.

Resolutely yours,
XXXX

Testing Collected Feeds as an Embed

2008-08-14T23:13:00.000-07:00

Trying to see how hard it is to collect everything I write in one place. If you can see a window with a whole lot of stuff in it, then it works :-)

Everything as a Service Podcast

2008-08-14T22:57:00.000-07:00

We did a live web radio broadcast last week on SaaS last week that I forgot to blog about. Eric Kavanagh (you may remember him from the TDWI webcast program) has branched out into live online radio sessions and they're fun to listen to. There have been a bunch of interesting topics on DMRadio, including column store databases, virtualization, dashboards and pervasive BI. It's better live since you can ask questions, but the archives are available.

Beware of Sit-4-Less Misleading Adverts

2008-08-06T18:16:00.000-07:00

I spend a lot of time at my desk, so I bought a decent Aeron chair from Sit-4-Less. Don't make the same mistake.

When they say a chair is "fully loaded", it doesn't mean the same thing that it does to Herman Miller. Ideally, you get all the various adjustments plus some sort of back support. Sit-4-Less appears to strip the back support and still label the chairs "fully loaded", without discounting for that fact.

They also shipped me a torn up box with an allen wrench taped to the inside and no screws. Makes it kind of hard to sit in since there's no way to attach the back, minus pieces of course. Their first response was "Did you lose the screws?" The best way to establish rapport with the customer is definitely to go on the offensive. I have screws coming, but they say that they won't include the back support, one of the reasons I ordered the damn thing in the first place.

Looks like a dispute on the credit card charge until they take the return or refund me some money. Bottom line: buying on eBay doesn't look so bad any more.

Update: never received screws. AMEX dispute shows a slipe to some guy in Oklahoma. I hope he enjoys the gift of chair parts.

Update 2: still no action

Update 3: 9 months later, I receive screws! Obviously they've come up with a unique way to create chair parts requiring a human surrogate mother. I wonder how long for the rest of the chair? Perhaps its elephant surrogates for the bigger pieces. I hear elephants gestate longer.

Live Online Radio: Emerging Technologies for Information Delivery

2008-03-27T22:57:00.001-07:00

DMRadio has a live show on Friday, March 28 at noon Pacific, 3:00 PM Eastern, about emerging trends in the information market, including open source, data and web 2.0. It's live, meaning you can ask questions of the panel, unlike regular podcasts, so have at it.

The lineup is excellent, and not just because I'm on it. The other people will be Dan Gisolfi of IBM, Chris Keene of Wavemaker and Doug Moran of Pentaho. What I find most interesting is that we're all talking about emerging technologies, and every one of us has an affiliation with open source.

Here's the description for the broadcast:

Convergence, Emergence – Part III

Mash-ups, composite apps, SOA and open-source – these emerging trends present game-changing possibilities for the information manager. Which companies and technologies should you track? Tune into DM Radio’s next broadcast: An Information Revolution, Part III, to learn from some of today’s brightest stars: Dan Gisolfi from IBM’s Emerging Internet Technologies group; Doug Moran, VP of community and co-founder of Pentaho; Chris Keene, CEO of Wavemaker; and special guest host, Mark Madsen of Third Nature.

In parts I and II, we talked about convergence. In this segment, we’ll talk emergence. Topics covered will include: what constitutes a mash-up, and why it can be so powerful; what “situational apps” are all about; how much traction open-source is really getting; the “other shoe” dropping in the open-source evolution; how AJAX can streamline creation of Web apps, despite its single-threaded Achilles heel; and how the composite app might just unravel the enterprise software licensing ball of yarn.

If you miss it, you'll be able to download an mp3 later. Check it out.

The Luther Burger

2008-03-19T11:28:00.000-07:00

Browsing can lead to some odd discoveries. While looking for banking information, I came across the "Luther Burger" in my search results. What's a Luther Burger?

"A Luther Burger is a hamburger, specifically a bacon cheeseburger, which employs a glazed donut in place of the bun."

This is a good explanation for why Americans tend to outweigh almost every other nation, even as we are the most diet-obsessed. Found via Real Life Comics.

Open Source DW Meeting in SF This Week

2008-02-26T14:28:00.000-08:00

There's going to be an open source session with some presentations, demos and networking at the Silicon Valley TDWI chapter meeting on Thursday, February 28 from 2:00 - 5:30 in San Francisco. This event is open to the public (and it's free!) but you do need to register first.

The announcement and agenda is listed below:

Attention All Data Warehouse and Business Intelligence Professionals in the Silicon Valley Area

We cordially invite you to attend our upcoming TDWI Silicon Valley Chapter meeting on February 28, 2008. Come meet other local BI/DW professionals, swap business cards, share ideas, and exchange career advice while listening to quality presentations in a vendor-neutral setting, which is the hallmark of TDWI events. TDWI Chapter meetings are open to all BI/DW professionals and are FREE of charge. In addition, when you attend a TDWI Chapter meeting, you are eligible for a 10% discount towards a new, renewed, or extended TDWI Membership. Don't forget the prizes! At each event our sponsors graciously donate fantastic prizes. Past giveaways include Apple iPods, TVs, DVD Players, and more. Please see the agenda below.

When: Thursday, February 28, 2008, 2:00 – 5:30 p.m.
Where:
A.P. Giannini Theater at the Bank of America Building
555 California Street, San Francisco, CA
(Theater is on the ground floor off the California Street Entrance)

Agenda:
2:00 - 2:25 p.m. Mingle with your colleagues over light refreshments
2:30 - 2:45 p.m. Welcome, Chapter Intro, TDWI Goals
2:45 - 3:15 p.m. The State of Open Source BI - Mark Madsen
3:25 - 3:45 p.m. Vendor Demo - JasperSoft
3:45 - 4:05 p.m. Vendor Demo - Talend
4:05 - 4:20 p.m. Networking Break
4:20 - 5:15 Panel Discussion on Open Source Software moderated by Mark Madsen. The panel includese professionals from JasperSoft, Pentaho, and Talend as well as other users of Open Source technology.
5:15 - 5:20 p.m. Close

Time for Open Source Think Tank 2008

2008-02-06T22:22:00.000-08:00

The third annual Open Source Think Tank hosted by the Olliance Group and DLA Piper starts tomorrow in Napa Valley, CA. This is an invitation-only event for leading open source thinkers to collaborate on ways to resolve commercial open source issues. I'm looking forward to the sessions, though I wonder if discussion will be dominated by the MySQL and Yahoo acquisitions. It's not off-topic, but I'm hoping that things stay on track for the primary topic, the future of commercial open source, rather than current mergers and acquisitions.

One problem I've seen with lots of writing about open source companies is that somehow, because it's open source, it must always be non-profit. While I personally believe there should always be a matching, freely downloadable version to what is offered under subscription, I don't think that means you can't sell closed software to supplement the open software. "Commercial" and "open source" are not opposite ends of the spectrum, they're orthogonal.

I've been seeing resistance to open source adoption above the infrastructure layer. Lots of reasons having to do with FUD, but some good valid ones too. Skillsets is a big one.

The thing is, why is skills a problem, when we find skills for other development / IT jobs? I think this is a symptom, and the real disease is the lack of finishing in OSS applications. Some are good, but many others involve several packages, and there are version incompatibilities, and you quickly run into problems that require knowing Apache + Tomcat or Jboss + perl or Java, etc. ad nauseum.

Heck, I've had that problem with using two different Business Objects products which under the hood deploy on open source technologies. It's pretty bad when even a COTS vendor is having troubles - their answer was, run these two applications on different servers because the Tomcat versions are incompatible.

Until the applications install and run as promised without worrying about dependency ugliness and conflicts, as well as polished documentation and troubleshooting manuals, I think OSS applications will stay in the underdog category.

Justice Scalia Supports Dictatorships Around the World

2008-01-16T14:31:00.000-08:00

It's mind-boggling the logic being applied in politics these days. Why was Scalia one of the worst supreme court appointees in history? How about this, after ruling against someone who refused to make patronage appointments repaying republican politcal favors in New York:

Justice Scalia and the other members of the high court were not persuaded by arguments that “one-party rule” effectively denied some people “a fair shot” at a judicial nomination.
“The reason one-party rule is entrenched may be (and usually is) that voters approve of the positions and candidates that the party regularly puts forward,” Justice Scalia wrote.

Yes, that piece of brilliant logic comes from a supreme court justice. Single party rule is very democratic. Working great in Pakistan. Hugo Chavez would agree. North Korea seems amenable to the concept. It's being demonstrated in practice in Burma. I'm glad the republican-appointed judiciary agrees. Maybe we can entrench that style of democracy in the US.

Unless they all get thrown in jail for fundraising for terrorists. That gives a whole new meaning to "strong on terrorism".

Network Solutions Again Pushing Legal Limits

2008-01-12T13:05:00.000-08:00

A couple days ago the news came out that Network Solutions (evil bastard domain registrar you should avoid like the plague) is swiping domains when you search. If you go to their site and search for a domain, they will automatically register it and then offer to sell it to you. They get away with this because of a rule that allows domains to be registered for 5 days without payment. They can take advantage of unsuspecting consumers and if no purchase results, free up the domain 5 days later. Register.com has been accused of the same thing in the past.

As proof, I did searches on a few and then checked them and sure enough, they were taken. Here are screenshots as proof. First, I searched a bunch of domain names. See how the Network Solutions site shows them as free (except one someone beat me to):

Then I went to GoDaddy and used their domain search and these same domains are all shown as taken, registered to Network Solutions:

If you want to cause them some pain, search for copyrighted or trademarked names and then notify the rights holders.

Retro-future Research

2008-01-10T22:50:00.000-08:00

I've been updating materials from one of my future-oriented presentations and came across this video on the future of cars from 1958. I think the only thing they got right was suburban sprawl.

Lots more of these retro-future videos on Youtube. It's funny how people were saying things like "It's useless. Who wants to watch kids on skateboards all day?" at YouTube's inception. Another example of poor predictions.

Diebold Voting Machine Hacks Demonstrated

2008-01-10T18:43:00.000-08:00

Black Box Voting has an article up about how bad the Diebold electronic voting machines really are. Three different hacks were described. To make it worse, some states like deservedly maligned Florida, have laws forbidding counting of paper ballots so there's no mechanism to look for manipulated elections. This company should be forced out of business. Let's hope your state isn't one using their machines.

Another Reason to Oppose Retroactive Immunity for Telecoms

2008-01-07T14:09:00.000-08:00

Because it would probably mean illegal government activities get grandfathered in as well.

Secret Underground Tokyo

2008-01-07T12:37:00.000-08:00

One thing we lack in most of America is mystery, like this supposed discovery of mysteries under Tokyo. Although it might land us in the land of Focault's Pendulum, which incidentally makes for a fun comparison to WTC conspiracy theories

Nominate Giuliani for This Kind of Performance?

2008-01-07T11:57:00.000-08:00

Great article in Discover on the effects of the toxic dust from the WTC building collapse. Yet another thing to add to the list of governmental failures. After reading this, it's hard to believe that anyone would consider Giuliani for any job, no matter how small.

Excel as a Database

2007-12-31T13:36:00.000-08:00

TDWI (particularly Wayne Eckerson) has always had a hangup about using Excel for BI, typically calling BI done in Excel "spreadmarts". While I don't fully agree that using Excel is as bad as people say it is, there are definitely some problems if it's not used in the proper context. So this link is for Wayne: how Excel becomes a database.

My 2008 Coolhunting Prediction: Goatees Are Out

2007-12-19T16:11:00.000-08:00

I'm an uncool person. I'm used to that fact. However, I have an affinity for cool people and cool things. That means that I'm often a leading indicator. The hard part is knowing what for, since I'm uncool. I once thought about doing work as a coolhunter but I don't have an affinity for the work, other than self-absorption. However, I've turned this to better use in the tech industry where I've been right the past few years much more often than I've been wrong (for a funny take on wrong, check IDC and the Itanic or Gartner and Qwest).

So on to my 2008 prediction: Goatees are Gone. I sported a goatee starting in the early 90's, where I was often frowned at by IT and consultants as some sort of freakish hippy. But around the mid-90's suddenly everyone had them. About two years ago I decided that it needed to go, and finally started shaving earlier this year, blending in again with the trendy web developer crowd. And now it's official: goatees are out (if it's made fun of in Dilbert then it's definitely uncool).

If you doubt my powers of prognostication, consider that I was right (2-3 years ahead) about the rise of coffeehouses, southwest cuisine, US wine consumption, sushi, ecotourism, commercial blogging, bold home and restaurant color schemes, the non-starting of natural language search, online video sharing and square dinner plates (which you can now get anywhere but which I had to search for weeks).

I was also way off on real estate investment in the bay area in the late 90s, Microstrategy dying after suing one too many customers, and MDM products, conflating them with Modern Drunkard Magazine and giving them (not the practice but the products) a pass. Never claimed to be perfect.

TripAdvisor Travel Site Updated

2007-12-19T15:12:00.000-08:00

Out of all the hotel sites, I've found TripAdvisor to be the most helpful when looking for hotels in places I haven't been. They seem to have more hotel reviews than other travel sites. I often use review sites when I'm unsure of the neighborhood, general location, etc. Reviews here have saved me from staying at some dirty, nasty places (I looked in on a few that I passed on recently).

They've been adding features over time and done some site updating. One feature I played with was the "where have you been" map, which I'm sure is all over the web. It was a little slow on adding pins, but I had fun thinking of all the places I've been and have yet to go. It seem like a lot until I looked at all the blank places on the globe:

Visit TripAdvisor.com

How Do Get Better Speaker Ratings

2007-12-13T14:37:00.000-08:00

I'm at conferences a lot, and I listen to a fair number of webcasts and podcasts. My one piece of advice to presenters is get to the point. Today I listened to 25 minutes of a 1 hour talk on what should be an interesting topic. The first 9 minutes got me through the introduction and landed me at the "agenda" slide. Several minutes on agenda got to the actual topic intro, and a few minutes of this finally got to the presentation material I wanted to hear. 25 minutes of that and it was Q&A time. 14 minutes of wastage to 25 minutes of content is a very poor ratio.

This is why I never listen to webcasts live. I can skip all this crap in recorded webcasts and podcasts.

To speakers: I listen so I can learn. I do not listen so I can hear about your career history or how wonderful and super-special your company is. I don't listen to learn about travel, hotels, your mother's bursitis and particularly your pets and children.

I rate people very low when they waste my time (and I know others do the same to me). Here are some things to avoid that will help you to get better ratings:

Minimize personal introduction - Who you are, where you're from, and a small amount of credentials for background is all anyone needs. If we want more we'll Google you, ok?
Minimize agenda - We don't need a detailed breakdown of what will be on every slide in the talk. Show the agenda, hit a couple highlights, and get to the talk. Total time for these two things should be no more than 2-3 minutes, tops.
Ignore the "tell them what you're going to tell them, tell them, tell them what you told them" advice - It's bullshit. Agenda and a summary are fine for this. Don't tell me the same crap three times or we'll think you're suffering from a head injury.

Really, just getting to the point will put you well ahead of the other guys. You know who the worst offenders often are? The people who should know better: industry analysts like me. It's just that most of us are prima donnas who can't be bothered to read our speaker ratings.

Must Be Something in the Water: "open season" on Industry Analysts

2007-12-12T16:00:00.000-08:00

I've noticed a significant uptick in Gartner criticism over the past 6 months. From little offhanded remarks at conferences (like mine when looking at a couple of their quadrants related to integration), to Colin's statements about their relevance, to this, the strongest reference I've seen in print:

"No show ripping on Novell and Red Hat would be complete without a helping of cynicism directed at Sun Microsystems, Facebook and the masters of the felch spoon over at Gartner."

...which is part of the description of the Open Season podcast this week. Trash-talking is always entertaining.
Now, I happen to know some of the Gartner analysts and like them. I also know some that I'd like to bang repeatedly with a shovel. But as an organization, I don't think it offers much of a service to the industry given how products and technologies are evaluated and how those evaluations are paid for. The problem is that if Gartner went away today, some other company would soon become exactly like they are.
Reality in the analyst world is that there is a love-hate relationship between vendors and analysts, and there are constant ethical challenges and real or perceived conflicts of interest that you have to watch out for. The slow erosion of the profit motive over telling the truth leads to the current trust situation, in the same way it has worked to destroy the journalistic tradition in so many countries, particularly the US. If anything, I look at some of the analyst firms the same way I look at Fox "news".
Analysts should be like investigative journalists. Often they are forced to soft-pedal negatives by their managers to avoid jeopardizing revenue. More insidious is self-censoring (like the white house press corps or the celebrity TV interviewers) where negatives or embarrassments are avoided for fear of losing access. OMG! Tom Cruise isn't talking to Gartner and went to Forrester instead! Except in this case Gartner is like an important tabloid you don't want to anger.
What the market really needs today is an analyst firm that tracks analyst firms, rates their predictions, examines their conflicts and assesses the quality of their work. Maybe the Open Season can create an annual "felch spoon" award too.

Open Source Channel at B-Eye, Funny Kindle Review

2007-12-06T17:15:00.001-08:00

The open source BI channel is up and running at the B-Eye Network. I'll be collecting resources, papers, sites as well as blogging about open source over there, so there won't be many OSS posts in these parts any more. Other BI-related or emerging tech content is landing over at Intelligent Enterprise now. Plus there's this whole new emerging tech site I've been working on that hasn't launched yet. I've been slowly distributing myself across the Interweb. I hope one day to dissolve into the network like that dude in TRON.

That mainly leaves items of interest like this great review of Amazon's Kindle ebook reader , killing all the annoying hype about the latest device that won't take the market by storm because of brain-dead EULAs and DRM. It also leaves this as the place for snarky comments and things that aren't safe for work so would get pulled from the other sites.

Considering a Mac

2007-12-05T13:59:00.000-08:00

I need to get a new laptop. The horrors of Vista are scaring me into considering a Mac. I don't use one now even though I spent several years as a Mac developer because all the enterprise software I look at runs under Windows. Turns out I don't need to travel with most of that, so I can be unchained. This video turned up as I was poking around looking for a devil's advocate.

Who's Going to Buy Whom Next Year

2007-11-20T02:22:00.000-08:00

It's the time of year when everyone starts talking about what's going to happen in the market next year. While hanging out after the sessions in Amsterdam, we all got to talking about this. Most agree that nobody is going to buy Microstrategy, IBI or Ab Initio any time soon. Some talk about the BI players who are left, mainly Actuate.

I've been wondering what HP is up to. They bought Knightsbridge and introduced the Neoview, but they haven't got anything else going on. IBM, Oracle, Microsoft, SAP all have data and application integration software. HP doesn't. Would have made more sense for them to buy BEA than for Oracle to. Oracle's buying strategy hasn't made much sense lately. It's like they want to be the ark of software and sell two of everything.

Everyone wonders how long Informatica, Actuate and Teradata will stay standalone, even though TD just separated from NCR.

It looks like all the main BI vendors are going to become application/technology stack elements. The bet is that a couple years from now BOBJ will be for SAP customers, Hyperion/Siebel/etc. will be for Oracle apps (Fusion) customers, Microsoft for Dynamics users, and Cognos for people who are playing the enterprise Java stack. The enterprise stacks (.NET, Java, SAP) are so complicated it's almost impossible not to shove BI functionality into them.

Scalzi Reviews the Creationism Museum

2007-11-13T17:31:00.000-08:00

Writer John Scalzi has written a really entertaining review of the Creation Museum. A good antidote to the depressing propaganda from the other day.

"Imagine, if you will, a load of horseshit. And we’re not talking just your average load of horseshit; no, we’re talking colossal load of horsehit. An epic load of horseshit. The kind of load of horseshit that has accreted over decades and has developed its own sort of ecosystem, from the flyblown chunks at the perimeter, down into the heated and decomposing center, generating explosive levels of methane as bacteria feast merrily on vintage, liquified crap. This is a Herculean load of horseshit, friends, the likes of which has not been seen since the days of Augeas."

Finally, Truth in Advertising

2007-11-12T12:33:00.001-08:00

Consumer Reports is correcting some misleading TV advertisements. This first video deals with "restless leg syndrome", and it's about time. I'm sick of pharmaceutical ads aimed at specific disease that make it seem like we all might have them, while showing drugs that have side effects far worse than the disease itself. Gambling? Intense sexual urges? Whaaa?

They're just side effects, so they must be minor. Drugs can solve everything! Buy more drugs! Maybe we could use them to solve the homeless veteran problem too!

Veterans Day: So Much for Supporting the Troops

2007-11-12T12:05:00.000-08:00

More than 25% of the homeless population in the US are veterans. Yet veterans make up only 11% of the US population. That's a big disparity and points to the need for the Republican administration to do something to solve the problem.

The Republican party has long shouted the message that "welfare mothers and homeless should just get a job", using it as a rationale for slashing the safety net, preventing passage of medical care reform, and cutting back on services for anyone who serves in the military. Meanwhile, companies like Haliburton, Black Water and KBR defraud the government of tens of millions and face no punishment, only blocked investigations by their political cronies

According to this article, there are roughly 196,000 veterans with nowhere to go, many of them veterans from Iraq versions 1 & 2. This is pretty pathetic for a government hell-bent on using the military wherever possible and calling anyone who opposes them "unpatriotic". I think "patriotic" would mean supporting the people you ask to die for your ideals after they come back from the war you threw them in.

How many speeches will we see today from politicians about how proud they are of veterans while these same politicians stab them in the back? Yes, that's right, veterans were not allowed to march in a Veterans Day parade.

You can do something about this. Vote the bastards out of office and get some people in who give a damn about governing, and help out a homeless shelter with a donation. It's the only safety net left to many of these veterans and they deserve our help.

SOAP and WS-* : This is Never Going to Work

2007-11-09T20:52:00.000-08:00

Not going to work. May as well call WS-* Death-* since that's where it appears to be headed. I've had my head in the sand with "enterprise" web work, doing REST APIs and such, so I surfaced a couple months ago and looked at all this WS-* SOA stuff and also Java stacks/platforms. They make my brain hurt.

All this reminds me of the days of CORBA and object request brokers and how the world would be saved by all this magical interoperability. After a dozen years it almost worked, so we moved on to SOAP which begat WSDL which begat UDDI and together they spawned WS-KitchenSink.

We now have WS-I: Web Services Interoperability. A standard to define interoperability of standards in the WS-* arena because the standards aren't standard enough. Joy! Sure is reminiscent of the early 90's when I gave up on C++ and transaction processing monitors and CORBA and focused on databases and data integration.

I'm glad I've been ignoring the enterprisey web services stuff and paying more attention to how real work gets done in a web 2.0 world. Hopefully the RESTian people will borrow the good bits of WS-* and move everything forward. Seems to be working well enough outside the enterprise where issues WS-* is supposed to address are potentially tougher than inside the enterprise.

Future of Business Intelligence Is Coming From Outside the BI Market

2007-11-02T17:18:00.000-07:00

My keynote for the Orlando conference was all about the future. Instead of trying to predict exactly what we would be doing in five years' time, I talked about why it's hard to know that, why the people who do it (us industry analysts) are so often wrong, what we can look at to guess the shape of the future in our market, and some examples of smart people doing work that's already in the next generation of data warehousing.

The slides are posted below. I'll have a transcript posted once I have it cleaned up for public consumption.

The future of business intelligence is most likely going to come from (or be heavily influenced by) consumer web sites, consumer electronics and games. I'm not seeing a lot of real innovation in this market from the major players. They seem to still be adjusting to the architecture change from client-server to basic web 1.0.

At least I know some of my assumptions about the future are right. Cindi Howson mentioned that most customers are still running desktop versions of BI software, so the BI vendors are still having to pay too much attention to the past. While sharing a taxi with Claudia Imhoff I checked my thoughts about the younger generation against her experiences with her daughter. Surprisingly, I got a few things right. She also had some great observations about what people need versus what they want and what IT gives them. See her blog for of her thoughts.

| View | Upload your own

Periodic Table of Marketing Elements

2007-10-22T12:50:00.000-07:00

I like these periodic table visuals. This one on marketing is well-done. Hover over an element and it provides the description for you in the central box.

Built by Kolbrener, a branding agency

Customer Service Hammer Time

2007-10-18T10:55:00.000-07:00

Mona, a 75 year old woman, got mad enough to go to the Comcast office and whack on some computers with a hammer after she had a series of problems, went to the office, waited two hours and had the manager sneak out the back door. Sweet.

After I hit the keyboard, I turned to this blonde who had been there the previous Friday, the one who told me to wait for the manager, and I said, "Now do I have your attention?"

This popped up after I'd spent the past several days outlining customer service measures and the relationship between customer service and business performance. The Mandalay Bay in Las Vegas earned a lot of enemies via lousy service over the past two weeks.

I've wanted to do take a hammer to Delta (possibly the worst airline in the US). Interestingly, someone did a survey and showed that one of the simplest things an airline can do to raise customer satisfaction is to give them a couple more inches of seat space. Next to getting you where you want to go on time. And not lying or screwing you when there's a problem.

US Government Plane With 3.7 Tons of Cocaine Crashes in Mexico

2007-10-14T12:23:00.000-07:00

A US plane with 3.7 tons of cocaine crashed in Mexico - a plane spotted at Guantanamo Bay and suspected of being used to transport prisoners. i.e. a CIA plane.

The highly successful "war on drugs" must be going smashingly, since the Bush administration appears to be importing more opponents. Nothing like a dummy US company bringing in drugs under the protection of the CIA. This way we can keep up the supply so the hard-working state and local police have someone to arrest.

No wonder Mexico is getting fed up with US complaints about drug imports. Our own government is doing it to us, just like they did during the Iran-contra mess. If you haven't looked at the Lessig's lecture below, now would be a good time.

Lessig Talks About Corruption

2007-10-14T11:31:00.000-07:00

As a followup to the last post about illegal surveillance, here's the new Lessig lecture on how to fight corruption in government.

If you don't know Lawrence Lessig, he's considered one of the top legal minds in the country and a likely supreme court justice at some point in his career, except that he chose to fight the mess of the US intellectual property system instead (e.g. bad patents, lousy copyrights, and a patent process overrun by greed and incompetence) because he felt that it was compromising our society and economy.

Now he's aiming at the bigger target of government corruption. I hope he becomes this era's Elliot Ness. We need one.

US Government Started Spying on Us PRIOR to 9/11

2007-10-14T11:05:00.000-07:00

This news is huge, and explains why all the telcos want retroactive immunity for breaking the law: former Qwest CEO says the NSA started illegal surveillance in February, 2001. What does this mean for the effectiveness of our government surveillance? This:

If the program was started in February 2001, then it completely failed to provide any information to help prevent the September, 2001 attacks.

Today, all this warrantless search and spying has been justified as a response to 9/11 and the need to "fight terrorism". But it started 8 months prior? And it was a complete failure?

In other words, illegal domestic spying by the Bush administration has nothing to do with terrorism, and trying to justify it after the fact in light of this total failure is a lie. Now we're on the path to getting a new attorney general who's as bad as Gonzales to help cover it up.

There are people in both parties who oppose covering up these actions and giving the telcos retroactive immunity, but not enough. The rampant corruption in this government never seems to stop and it won't if people don't hold their representatives accountable. Call your congresscritter and tell them not to give up more of your rights.

Holy Crap on My Mom's Shoes - SAP Buys Business Objects

2007-10-07T14:39:00.000-07:00

Rumors of Business Objects sale have been going around for years, including a decent one last Monday, although fingering Microsoft as a likely buyer is kind of thick-headed given Microsoft's BI announcements and product lines. That looked more like an attempt at misdirection than anything real.

They're calling this a friendly takeover, although I wonder how many employees will view it that way. With this purchase, the BI market now looks a lot like the ETL market. There's only one large independent vendor in each market - Informatica for ETL and Cognos for BI - and a bunch of mostly smaller companies left.

This is a great thing for SAP since they can now start taking in sales for BI where once it all went to third parties. They also get a decent set of data integration and data quality products to complement SAP's sore applications. It's good for Business Objects too, since this opens up the market for all those SAP accounts.

It's not so great for Business Objects customers long term. SAP isn't known for being a quick, nimble company or for offering reasonably priced products. The stand-alone relationship with SAP may prevent the worst of over-management from a large global corporation from happening. This doesn't mean much for customers over the short term since it shouldn't mean any major changes to existing operations.

This is definitely good news for stockholders since Business Objects said it will report earnings per share of 36 cents to 39 cents, well below projections of 51 cents per share [Reuters Estimates]. They likely would had a drop in share price with an earnings gap that large.

The acquisition is somewhat of a strategy shift for SAP since they have either built their own analytic tools or acquired relatively small companies. The purchase of OutlookSoft was a sign that this could be changing. The Oracle buys of Hyperion and Siebel and the Microsoft BI announcements were putting pressure on SAP since their BI story is comparatively weak.

I'm sure there are going to be lots of interesting questions on the briefing calls Monday morning, and probably a lot of low-content answers. If you feel like listening to the announcements, here's the call in information:
dial in number: +1 480 629-9564 (US), +44 207 190 1596 (UK), +49 695 8999 0701 (Germany). Replay number: +1 303 590-3030 (US), +44 207 154 2833 (UK); Replay passcode: 3792655.

Fall Events Lineup

2007-10-05T22:23:00.001-07:00

I'm headed to Las Vegas, first for the Teradata Partners conference, then for the IBM Information on Demand conference. Both look like great events. Two weeks at the Mandalay Bay hotel might be overkill though.

After that I'm off to Orland for the fall TDWI in Orlando where I'll do the full day ETL evalution session. This time around we have Informatica, Oracle and the open source company Talend doing demos. It will be an interesting comparison. I'm also doing the keynote on Thursday where I'll be talking about BI, the future and what a mess emerging technology and culture are going to make, with little history thrown in for fun.

Then the European TDWI conference in Amsterdam, followed by TTI's portal conference in Rome. The European lineup will be fun since I'm talking about open source BI and data warehousing, web 2.0, web data extraction and web scraping. Emerging technology is always interesting stuff.

Look me up if you're going to any of these events. Lots to talk about.

Silent Data Corruption, and Trips to Mars!

2007-10-05T21:58:00.000-07:00

Interesting post over at ZDnet about silent data corruption, the kind where bits shift undetected. That means you've got bad data, or a bad file, or a bad file system. The bad news is it's worse than expected. The really bad news is that like anything, entropy is going to increase over time. I can't wait to see how this affects the government's data analysis initiatives and the no-fly list.

Some other great sciencey news:
Nanowire for faster, denser, persistent storage (from the same blog). Too bad it's 10 years out. That does me no good right now. I just hope this isn't another niobium crystal technology that ends up not working in real life.

Breakthrough in photonics means we might get to go to Mars after all. I want this one even sooner than the nanowire memory.

Mashups and Shopping By Color

2007-09-16T13:17:00.000-07:00

At the TDWI Executive Summit I talked about the web mashups, BI and the blurring between the two. One of the mashups I showed was ColorPickr, a nifty app that pulls images from photo site Flickr based on your choice of color from a palette.

I use this as an example because the typical reaction from the IT audience is "Neat. But it's not really useful for anything." You can come up with a half dozen right off the top of your head if you've worked on a web web site, for example finding images that match the site color scheme.

I use a more compelling example to emphasize that you're focusing on the wrong thing if you see mashups as useless. Think of the last time you shopped online for clothing. What if you want a blue shirt?

One of the features of all shopping sites is what the web folks call "faceted search" allowing you to narrow choice based on product features. In data warehouse terms you can think of this as a constraint on attributes in the product dimension.

Yet none of the shopping sites I'm familiar with take advantage of presentation opportunities the web offers. They still merchandise online in the same way they lay out shelves in a store. So you filter by type of clothing, style, gender and size. No way to search by color. That would be a crazy way to organize shelves in a store, but there's no reason you can't do this online. It can even make the experience more fun.

Today Yahoo! introduced a new "shop by color" feature on the Yahoo! Shopping site. Exactly the sort of thing I was talking about.

The problem most people have is one of perspective. Many online mashups and the ideas behind them are useful well beyond the original concept. One of the things I like about mashups is that they evolve from a sense of play as much as from a specific goal. This is a creative element that's been lacking in many IT shops. The web 2.0 crowd bring some of the fun back to working in IT.

Web 2.0 Moves into IT, Goodbye RIA as a Name

2007-09-14T16:44:00.000-07:00

There are tons of web 2.0 companies out there who were trying to either create consumer sites or provide technology to other consumer/business facing sites. I expected them to begin shifting focus as they run out of places to sell to, the market becomes overburdened by web 2.0 technology providers, and companies starve for cash.

One of the areas with a lot of vendors is the user interface arena, generally categorized as Rich Internet Application (RIA). I'm already seeing signs of repositioning in this overcrowded market. There's only so much money to be had before the need to sell to traditional IT. What usually happens is companies try to re-market into IT and discover their tools aren't sufficient. I suspect next year will be a great year for acquisitions and shutdowns of RIA companies.

I've spotted a few who changed colors from "RIA" to mashups. Jackbe is a good example, but they at least started out as "Rich Enterprise Application" provider aimed at IT instead of a more typical RIA play. Check out their site from last year, then look at this year, and you see the same products with different labels attached. I've talked to a few other companies in this space making the same shift in marketing. I think the other guys will have more trouble because they didn't start with the goal of selling into IT, and that's a different beast from VC, word of mouth and wishful thinking.

Time to say hello to "enterprise mashups". With major vendors plowing money into this space (Microsoft, Adobe, Google, IBM, BEA) you can expect the marketing around mashups and trying to link them to SOA to be one of the marketing pitches you'll be subjectd to in the coming year.

Five Minute Data Integration Podcasts

2007-08-31T12:53:00.000-07:00

Podcasts we did on defining data integration technologies just went up at SearchDataManagement today. Each describes a particular technology, what it's used for and when its appropriate. We did four of them on ETL, EII, EDR and EAI, with plans to add several more on topics like master data management and data profiling. It's always strange to hear yourself as other people hear you.

Slides for Web 2.0 and BI Executive Summit Are Here

2007-08-24T16:49:00.000-07:00

The panel session on BI futures at the TDWI Executive Summit was a fun way to wrap up the day. I like to see the heavier content lighten up before breaking for a reception.

Since half the presentation was demos on web sites and mashups, and a quarter was talking, the slides are here for posterity and as a reminder for people who were in the session. The links/mashups are embedded in the slides so you can check them out for yourself.

Slides for Data Federation/EII Uses and Abuses

2007-08-04T14:20:00.000-07:00

I just uploaded the slides for the data federation/Enterprise Information Integration uses and abuses webcast to SlideShare (embedded below). You can also follow this link to the archived webcast if you want to listen to the talk. Minimal text on the pages so that is probably your best option if you want the details.

Podacst Interview With MashupCamp Developers

2007-08-02T13:47:00.001-07:00

IBM put up a podcast of a group discussion about mashup development using QEDwiki that was recorded after Mashup Camp. Scott Laningham from IBM developerWorks interviews Kenny Spade, Yuval Tarsi, and Renat Khasanshyn and me. A vidcast would be better since it can be hard to get a sense of the mashups with only audio.

Readers Berating CIO Magazine for Layout Equivalent of Spamming

2007-07-26T15:14:00.001-07:00

I had more fun reading the comments on this CIO Magazine article than the article itself. In fact, I didn't finish the article because it was the thing that pissed all the readers off.

Lesson to learn: sure, you may be trying to maximize ad revenue, but dropping the ad to content ratio to 1 ad per 100 words is taking it too far. They have 19 (!) pages of article, one paragraph per page. The side effects are damn funny, though. The comment paragrpahs count runs 5:1 over the article paragraph count.

RSS Sucks For Mashups, But It's Ubiquitous

2007-07-25T00:09:00.000-07:00

Another thing we learned while banging away at web sites to get data was that RSS stinks for slinging web data, yet the tools available don't deal well with Atom feeds. Atom is great for this purpose, but hard to locate tools for (other than code libraries). Being both ignorant and lazy, I'd prefer to integrate tools rather than write 3x that much code in PHP and Python.

Our first attempt to grab data from Salesforce.com generated an Atom feed. We did that because we wanted to get a bunch of data and feed different elements to other extractors to collect data. Sadly, none of the tools we were using could consume Atom. We had a choice of programming a custom widget for parsing Atom to get the data we wanted or switching to RSS. We did the latter because it was expedient given the time constraints.

The problem with using RSS for data is that it only has a few fields to store data in. We ended up using the title, description, link, and even the date fields. This meant having to take 8 fields of extracted data and publish two separate RSS feeds, or stuffing multiple values into a single field like description. Luckily, we didn't need some fields broken out so stuffed things into fewer fields. Still, I had something like nine RSS feeds going at one point.

The other side was getting feeds. Lots of sites generate RSS feeds, but no Atom. So we're stuck with crappy feeds or we do it ourselves, but then have to DIY the consuming side as well. Let's hope Atom usage picks up in the future.

Web Data Sourcing Tools We Used for the Mashup Contest

2007-07-24T09:02:00.000-07:00

We just finished recording a podcast for IBM DeveloperWorks which will be up in the next few days so I was looking to see what else happened at Mashup Camp while we were writing code. We got a mention at Programmable Web (first place I go to see what's new in the mashup space). I wish we had been able to spend more time in sessions, but being heads-down in new tools was still worthwhile.

Apart from the possibility of winning prizes, this is the best learning environment I've found for this space. Unless you spend a lot of time reading through blogs, you aren't going to find many resources. Besides, nothing beats learning from other people while doing.

Here's the rundown of tools Renat and I worked with:
QEDwiki (not an official product or downloadable, yet)
Apatar
Dapper
Kapow
Yahoo! Pipes
and lots of Google sources

One thing that hasn't been mentioned in most of the news is that all these companies had people at Camp. To be honest, if it weren't for Dan Gisolfi and Meg Sorber from IBM, we never would have finished work by the time the event closed. They stayed up late to help us with problems, bugs and techniques using QEDwiki.

I used Dapper a lot to scrape pages and make RSS feeds. I ran into problems with Dapper and the bad HTML practices of some web sites. Fortunately, Eran Shir and Jon Aizen (the CEO and CTO of Dapper) were there to help out.

Out of all the things I've worked with, theirs is the most impressive because of its simplicity. Unlike Pipes, which manipulates RSS feeds, Dapper scrapes pages and turns them into feeds in many different formats. Dapper + Pipes is a great combination. Kapow is an industrial strength scraper, so Dapper is not as powerful as Kapow's tools, but it's a lot easier to use for something quick and dirty.

We used Apatar because it was the only way short of directly coding to APIs to get data from Salesforce.com. And it's open source. And Renat knows how to use it. It's not a page scraper, it's a data integration tool, so it does things these other tools can't. Overall, the combination of a DI tool, a scraper, and a manipulation and delivery formatting tool are what you need to get data for mashups if you're doing it inside an IT shop.

QEDwiki is the assembly hub, so it doesn't provide data sourcing or manipulation features. IBM is going to include another tool in the kit for that. They did a demo of this during Mashup U, but didn't have it available for us.

Watch a Walkthrough of Our Mashup Camp Contest Winner

2007-07-23T23:08:00.000-07:00

Here's a link to coverage of the mashup contest winner (me and Renat Khasanshyn - not often I get to see "me" and "winner" in the same sentence) over at ZDnet: Mashup culture shatters crusty, stodgy old approach to business app dev

They did a nice job with the video of David Berlind interviewing me showing a walkthrough of the mashup and a little about what we had to do to build it. I'd comment on how intelligent that person doing the demo looks, but I'm too modest.

One thing that didn't get mentioned is that Apatar - which Renat used to pull a data feed from Salesforce.com - is open source. It's free as "freedom" and free as in "beer". Hard to beat that.

Sadly, no walkthrough of my answer to the Starbucks finders on Programmable Web, the Starbucks Anti-Finder (on the right). Given how many stores there are, I think the real challenge is showing a map of coffeehouses that aren't Starbucks, so I created a Starbucks locater that can't find Starbucks, even if you were to feed the lat/long position to the map. You're welcome, Brian - now you don't need to see corporate coffeehouses. Version 2 will exclude all public corporations from the list, making cafe hunting even easier for the anti-corporate crowd.

And no, there is no such thing as Jose Cuervo cereal.

Getting Data for Mashups

2007-07-21T22:31:00.001-07:00

Getting data is still the hardest part, even with great tools like Dapper and Kapow. They aren't perfect, but neither are the web sites they go after (see previous post about Yahoo!). One thing we learned building the main mashup for the competition was that it's helpful to have an ETL tool around. We used page scrapers and RSS mungers and the like, but the only way to get data from Salesforce.com is to write to their API.

That's no fun. Luckily, we had access to Apatar which has prebuilt connectors for Salesforce, so we were able to go in and fetch the data we wanted without having to resort to client-side javascript calling Salesforce APIs. To do that would have meant coding custom widgets in order to address communication between UI controls, ad there was no way to do that in he time we had.

I love open source. Amazing how quickly OSS data integration has been coming to market. Aside from Apatar there's Talend, Kettle (Pentaho Data Integration) and SnapLogic (though they aren't technically an ETL tool). I saw a mention of JitterBit recently but know nothing about them yet.

Unexpected Lessons From the Mashup Camp 4 Contest

2007-07-21T21:09:00.000-07:00

Mashup Camp is done and we came away with 4 of the prizes from the IBM (and Dapper, Kapow, StrikeIron, Accuweather) mashup building competition. Not a clean sweep but we did our best.

While sourcing information for the mashups, one thing really surprised me. Yahoo! generates really poor HTML that's hard to parse. We had identified data for which there were no feeds in several areas (like Yahoo! finance) and built scrapers. We couldn't get any of them to work (I tried from five different sets of pages). We ended up going to several other sites like Google finance to get our data. Googles sites were simple to scrape, with no strange things going on.

Lesson learned: if you want to use Yahoo! go via one of their APIs or RSS feeds, otherwise find another source because you'll be pulling your hair out.

A related lesson is just how web-unfriendly Microsoft can be. I hadn't realized the difficulty of going after ASP-generated pages. If you use url-driven tools you are simply SOL. If you use a tool that can do directed spidering it's still not easy but works.

I'll be ready for the next contest. IBM gave two months of advanced notice, but neither we nor the #2/3 winner took advantage of the time. We didn't start until the day before the deadline, when the wine and beer provided by IBM encouraged us to help each other learn the tools to do the work.

Mashup Camp 4, Day 1

2007-07-17T00:21:00.000-07:00

I'm at Mashup Camp 4 this week. Tired. Who starts a conference with breakfast at 7:30 AM and runs it until 6:00 PM? Based on today, someone who's wrangled lots of smart people to talk about leading edge technology.

Bottom line: lots of sexy UI stuff like Zude going on, less focus on the data integration, but more than I expected. Enjoyed the SnapLogic presentation, not so much because they are innovative (they are, but there are a half-dozen other entries in the general web integration market) but because they have a freely downloadable server-based open source offering. This circumvents one of the complaints I have about Yahoo Pipes, Dapper and OpenKapow: you can run your own server, where the others are services that host for you with no option of running your own server. (closed Kapow costs a lot, does more and runs inside your firewall).

Saw Google Gears demo of an application running online and off. That's some interesting stuff and it's still very early. Given the flakiness of the conference wifi, more people could have benefited from offline browsing.

Also excited because I just got my invite for the Google Mashup Editor so I can play around. So many things to try out, so little time. I'd love to integrate LignUp with one of the open source BI tools and build some fun voice-enable BI alerter demos.

Overall quality of demos and talks was high, with interesting things from both small startups and big companies like Yahoo, Google and Microsoft. Looking forward to tomorrow.

List of Data Integration Webcast Links for This Year

2007-07-09T11:58:00.000-07:00

Here are links to replays for all the webcasts I've done during the past year.

The Hybrid Data Warehouse: Extending the BI Environment with Data Federation
Webcast: 3/21/07

Challenges and Techniques When Hand-coding ETL
Webcast: 6/26/07

Extract-Transform-Load (ETL) Market Overview and Directions
Webcast: 6/13/07

Enterprise Information Integration Technologies: Common Uses and Abuses
Webcast: 2/14/07

Enterprise Integration Series: Evaluation Criteria for Selecting ETL Tools
Webcast: 8/2/06

Synced Video of Harvey Danger Song

2007-06-29T15:42:00.000-07:00

It's Friday. I enjoyed this video. You might enjoy it too. Particularly after about a minute. It's the type of thing that I miss about working in ~~the bay area~~ startups and young companies.

Lip Dub - Flagpole Sitta by Harvey Danger from amandalynferri and Vimeo.

Powerset Gives Natural Language Dead Horse Another Beating

2007-06-29T14:51:00.000-07:00

I was going to write about Powerset, a search startup / hype machine that keeps generating buzz without delivering anything, but Donna Bogatin delivered far better than I could with this post on their supposed coming out of stealth mode. I keep tabs on search because it's one piece of the information management puzzle, but in enterprise situations it's highly overrated and far more difficult to implement, despite all the "BI & search" hype.

Powerset hyped NL search queries, and now they seem to be hyping web 2.0 search. I'm one of those people who doesn't really believe NL search is that great a thing, for a lot of reasons I won't go into. But the mutating Powerset pitches and buzz marketing leave me clenching my teeth. I'm glad Insider Chatter was there to do the work for me. It's moved up on my weekly reading list. Here's her take on Google.

"The Google $160 billion plus market cap business plan is to CONTROL all the world’s information AND all the personal data of ALL the world’s citizens with the goals of:
1) selling high-priced advertisements against the content owned by others, and
2) data mining the personal information of others for its own corporate advantage."

Good take on Google's constant drumming of the "we're not so evil" mantra while they continue to slide into evil. Curmudgeonly. I like it.

Slides for the ETL Market Update

2007-06-27T18:01:00.000-07:00

These are the slides from the ETL market overview webcast I did for TDWI in May. Share and enjoy.

If the small fonts are hard to read you can go to the slideshare site (link on the lower right) where you can see a full-screen version of these presentations.
The replay of the webcast is now available at the TDWI website.

Slides for Hand-coding ETL Available

2007-06-26T17:06:00.000-07:00

I'm trying something different with the slides for this TDWI ETL webcast, using slideshare instead of the usual PDF and embedding it directly. I will convert and post a PDF later as I've done with all the other webcasts later. The downside of Slideshare is that the transcript doesn't contain the information from the notes pages, which makes presentations like this less useful.

Update: the replay of the webcast is now available at the TDWI website.

Navy OKs Open Source

2007-06-21T12:59:00.000-07:00

On June 5 the Navy gave the stamp of approval to open source software as an option for their contracts. This is a big deal since government procurement has all sorts of complicated requirements, and you have to be included in that process to get in the door. This paves the way for other government agencies to follow suit.

"Open-source software is now an official option for all information technology systems in the Navy and Marine Corps, according to a guidance memo issued June 5 by the Department of the Navy’s Office of the Chief Information Officer."

Good news for open source providers, as well as systems integrators. For the SIs, it means they can take a larger portion of the project budget for services, rather than having that money go to software partners. The question is whether they will pick up on that, given that they sometimes resell software and other times the software vendors get them into projects. It's a conflicted world out there.

How to Tell a Story With Data

2007-06-14T20:05:00.001-07:00

This is a great lecture by Hans Rosling, interesting and communicative visualizations, and he really knows how to show the story in the data. It's only 20 minutes and worth your time on two levels - learning something new and seeing someone good at presenting information.

You can play with some the tools he's using in his presentation at the gapminder site, as well as looking at the interactive 2005 development trends report I blogged about back then.

Helpful Resources for Scaling Web Sites and Applications

2007-05-31T13:28:00.000-07:00

There's a good collection of presentations about high-end scaling of web applications at Peter Van Dijck's blog. Most are hosted on slideshare, but there are some mp3s and PDFs as well. It's hard to find quality information on scaling applications of any sort. I used to deal with scalability a lot in the early days of the web. Technologies have changed quite a bit since then so I focus on scalability mostly in the data warehouse and business intelligence world now. I always pay attention when someone goes through the trouble of collecting a set of good quality material like this. I've only seen two of the presentations he's got listed, and never read the O'Reilly book he mentioned, which means more summer reading in the works.

Real Estate Data Visualization

2007-05-30T16:22:00.001-07:00

Trulia, a real estate search service, has an interactive map of US homes animated based on when the homes were built. It's fun to play with. The search service (the first link) has a well-designed navigation and display interface, better than others I've used. Worth looking at to see how information displays can be made interactive.

LOLTrek: This Meme Has Jumped the Shark

2007-05-29T21:31:00.001-07:00

If you don't know what this means, you probably need to stay in more. In this case, go find the "I can has cheezburger?" image, then visit a tribute to tribbles, LOLTrek. If don't know what "jump the shark" means then you're definitely having too much life and not enough computer and need to spend some time on Wikipedia.

Even funnier is Google correcting "I can has cheezburger" to "I can has cheezeborger" when I tried to track down one of the original images:

To finish off my news reading for today, "I'm in ur cellz killin ur mitochondrias" (Fanta screws with your mitochondria?) Good thing I don't drink those fizzy drinks.

Story About the First Computer

2007-05-29T09:52:00.000-07:00

The New Yorker has a story about the Antikythera Mechanism, possibly the first mechanical computer, built by hand in ancient Greece. I liked the conclusion best, that the device embodied the world view of an era, in much the way that our current world view is embodied in our own technology.

"One day in the spring of 1900, a party of Greek sponge divers returning from North Africa was forced by a storm to take shelter in the lee of the small island of Antikythera, which lies between Crete and Kythera. After the storm passed, one of the divers, Elias Stadiatis, put on a weighted suit and an airtight helmet that was connected by an air hose to a compressor on the boat, and went looking for giant clams, with which to make a feast that evening."
continue reading story

US Terrorism Data: Worst Practices for Data Quality and Governance

2007-05-26T13:52:00.000-07:00

The Department of Homeland Security is a perfect counter-example of how to deal with data. A way to learn good practices is to do the opposite of what the government does. Here's a list of worst practices to not follow from the DHS.

Worst practice #1: load all the data you can lay your hands on. Terror Database Has Quadrupled In Four Years discusses the "Terrorist Identities Datamart Environment" or TIDE, the source of data for airline, police, border and consulate watch lists. Their policy is to shovel everything possible in there, needed or not, because it might be useful some day. This runs counter to years of data warehouse practices.

"Each day, waves of new information are fed into terrorism-suspect databases... Ballooning from fewer than 100,000 files in 2003 to about 435,000, the growing database threatens to overwhelm the people who manage it."

Aside from this problem, do we really believe there are 435,000 suspected terrorists in the US? That's almost four times larger than our military in Iraq. What are they waiting for? They could take over the US now. Obviously something is wrong with all that data.

Worst practice #2: don't do anything about data quality. "The single biggest worry that I have is long-term quality control," said Russ Travers, in charge of TIDE at the National Counterterrorism Center in McLean." Yet he's doing nothing to QA the data before dumping it into the system. To make things worse, there's no way to remove or correct data once it's loaded.

"The bar for inclusion is low, and once someone is on the list, it is virtually impossible to get off it."

"In 2004 and 2005, mis-identifications accounted for about half of the tens of thousands of times a traveler's name triggered a watch-list hit."

"Sen. Ted Stevens (R-Alaska) said last year that his wife had been delayed repeatedly while airlines queried whether Catherine Stevens was the watch-listed Cat Stevens. The listing referred to the Britain-based pop singer who converted to Islam and changed his name to Yusuf Islam. The reason Islam is not allowed to fly to the United States is secret."

TSA can't tell the difference between a 70's musician and a senator's wife? That's a serious data quality problem. Given the 50% miss rate, it would be better for the police to toss a coin each time they arrest someone instead of consulting the watch lists.

The data management practices go one step further. They actually have a process to include names on the list that are no longer valid. Their idea is that if someone is dead (for example), then a terrorist might use that name since it won't be on the list. Perhaps all the dead people account for the growth of the database. Why not go one step further and put the names of everyone who died this year onto the list?

Worst practice #3: ignore the problems of your users.

"TIDE is a vacuum cleaner for both proven and unproven information, and its managers disclaim responsibility for how other agencies use the data. "What's the alternative?" Travers said."

Multiple agencies are complaining about wasted man-hours due to mis-identification. Airlines are routinely stopping people in airports, like Ted Steven's wife, even though the data is obviously wrong. God forbid that we should have any data management processes in place. Travers should probably get out to a data warehousing conference once in a while. Particularly since he said earlier that data quality is a problem. His current alternative of doing nothing isn't feasible.

But the problems don't stop with TIDE...

Worst practice #4: if the users aren't sold on the concept, build it and they will come.
Remember TIA ? Congress killed the program, so the people involved did it for the government of Singapore instead. Son of TIA: Pentagon Surveillance System Is Reborn in Asia tells how Snowden and company built a system for tracking people in a totalitarian state. Now they they want to sell the system they developed back to the US government. It's like doing an IT-driven data warehouse project, only with Orwellian overtones.

Worst practice #5: ignore the users. DHS has a monumental mess on their hands. Different government departments need different data, just like the finance department has different needs than the marketing department. Yet the DHS systems are being centrally mandated by (mostly) intelligence people with no idea of how other groups like the police or the border patrol need information. Combine this with poorly managed data integration and no governance and you have data being copied and misapplied all over the place.

A number of the articles I linked to came up via Bruce Schneier's crypto-gram mailing list. Always interesting reading, even if you aren't a security professional. Choice items from this month's issue:

"...tips on preventing terrorism" indeed. (Tip #7: When transporting nuclear wastes, always be sure to padlock your truck.)

YOU are big brother: Control and track your car from the 'net Or, as Schneier says, have someone hack into their web site and control it for you.

AMEX is Watching You AMEX has a patent application titled "Method and System for Facilitating a Shopping Experience," that:

"describes a Minority Report style blueprint for monitoring consumers through RFID-enabled objects, like the American Express Blue Card.

According to the patent, RFID readers called "consumer trackers" would be placed in store shelving to pick up "consumer identification signals" emitted by RFID-embedded objects carried by shoppers. These would be used to identify people, track their movements, and observe their behavior."

Breaches of personal data: blaming the myth and punishing the victim Can we stop blaming hackers for theft of information already?

The report states that "60 percent of the incidents involve missing or stolen hardware, insider abuse or theft, administrative error, or accidentally exposing data online."
...
Given that its data suggests that a significant portion of the blame should go to those who hold the data, the report argues forcefully for legislation that requires they meet minimum data safety standards.

Windows Vista code-signing to keep out evil spyware? I don't think so: VBootkit bypasses Windows Vista's code-signing mechanisms Microsoft spent a lot trying to secure Vista. So far, no dice.

Local Sheriff Suspects Al-Qaeda Or Teens

"This activity matches up with the M.O. of a terrorist casing a potential target," Steinhorst said. "It also matches the M.O. of a group of teens drinking beer and fooling around."

I don't read The Onion enough.

Feel Free to Carry Explosives on Planes, You Won't Get Caught

2007-05-26T12:13:00.000-07:00

I just got done with 6 weeks of continuous travel and I wasn't surprised to read that airport security missed 90% of improvised and hidden explosives during security tests, proving our system doesn't work. Yet we still have to take off our shoes, fork over our mouthwash and listen to "the threat alert has been raised to orange - report your neighbors". My favorite quote is this from a former TSA inspector:

"There's very little substance to security," said former Red Team leader Bogdan Dzakovic. "It literally is all window dressing that we're doing. It's big theater on TV and when you go to the airport. It's just security theater.
...
Dzakovic, who testified that the FAA ordered the Red Team to "not write up our findings," said the TSA is also trying to hide its results.
"The last thing TSA wants to do is look bad in front of congress and in front of the public, so rather than fix the problem, they'd rather just keep them quiet," said Dzakovic.

Sure would be nice the idiots running homeland "security" would stop trying to keep their evil overlords in power by pretending we're under imminent attack and instead focus on doing their jobs. I know that's a lot to ask of this administration.

Web Integration Talk in Las Vegas

2007-05-21T09:03:00.000-07:00

I'm off to Las Vegas for the Portals, Collaboration and Content Conference to give a talk on data integration for the web. Title of the talk is "Web Data Integration: Methods to Extract and Deliver Data for Portals and Web Applications." I'll be doing a run-through of integration architecture and technology choices to get data from where it is to where you want it. I had to cut back on the part that I find most interesting, getting data off web pages. Scraping, scrAPIs, RSS and the rest of the fun web stuff gets about 10-15 minutes towards the end. I'll probably repurpose the unused information into posts. The slides will make their way online some time in the next couple weeks. I can say one thing - they're pretty relative to some of the other presentations.

Microsoft Patent FUD Appears to be Exactly That

2007-05-19T13:03:00.000-07:00

Since I just finished running a day on open source at TDWI, I thought it would be worthwhile to comment on this. It's always hard to tell what's going on when they toss out FUD, but the authors of the study about potential Linux infringements say that Microsoft is misrepresenting their conclusions:

"The point of the study was actually to eliminate the FUD about Linux's alleged legal problems by attaching a quantifiable measure versus the speculation," he said. "And the number we found, to anyone familiar with this issue, is so average as to be boring; almost any piece of software potentially infringes at least that many patents."

This looks like another case of bogus interpretation similar to the Linux TCO study, where they concluded that Linux was ten times more expensive to run than Windows. It's true, when you look at what they compared: an Intel PC running Windows vs. an IBM mainframe running Linux.

Open Source BI Case Study Slides Are Posted

2007-05-18T20:06:00.000-07:00

We had a good open source session at the May TDWI conference. After I spent some time reviewing history, projects and adoption practices we got to the good part: short case studies, demos and a panel session with representatives from BIRT, JasperSoft, Pentaho and SpagoBI. I particularly enjoyed the panel session where we had some great insights from the panel on what's happening in this space and some of the problems people are facing. I also liked Cindi Howson (the most knowledgeable person on the subject of BI products I know) asking some basic but pointed questions that were on the minds of the mostly non-OSS audience.

Special thanks to Paul Clenahan and Jason Weathersby from Actuate/BIRT, Nick Halsey and Beth Mazur of JasperSoft, Lance Walter and Nicholas Goodman of Pentaho, and Grazia Cazzin and Daniela Tura representing SpagoBI. They did a lot of work and came at their own expense to represent their projects at this conference.

All of their overview and case study slides are available at Third Nature.

Profane Super Mario - I've Heard the Same Things From BI Users

2007-04-20T18:13:00.000-07:00

I came across this hilarious, profane 23 minute video walkthrough of a Super Mario Brothers level while browsing for material for a presentation on game mechanics and software design (NSFW without headphones).

My internal dialog when evaluating software is just like this, from the frequent bouts of swearing to the shouts of triumph and shrieks of despair:

"This is worse than reading an RL Stine book. This is worse than reading YouTube comments! This thing sucks donkey ****!"

I've been in the office with a great many very smart, non-IT people using a BI tool for the first time. I see their faces twist and grimace, watch them turn shades of pink, see the sweat bead on their foreheads and drip on their keyboards. I know exactly what they think about the developers at that moment: "They were too busy watching Magnolia and cupping each other's balls."

I review and use a lot of enterprise software and "usability" is not a word I associate with most of it. I provide occasional toned-down feedback to vendors when I really want to track down and throttle the "usability experts" they hired to give them advice, usually people with zero domain expertise. I refrain because I'm afraid of choking off business.

I thought it would be an interesting idea to take this shared pain, go back to things that work and are engaging or infuriating in game design, and bring some of the design lessons to the application audience. Working with software, particularly BI tools, ought to be enjoyable, or at the least satisfying. The one-off mashups done by web 2.0 developers are sometimes a better model for BI than the approach BI vendors are taking today. At least the mashups make looking at data fun.

Addendum: this chart of profanity use by cause

Redo: Open Source Talk at Portland DAMA

2007-04-15T18:23:00.000-07:00

I'll be back in Portland this week to give a talk on open source in data warehousing. The last one was cancelled due to a snowstorm (scroll to see video). Hopefully this one will fare better.

The talk is an abbreviated version of of the session I'll be running at the Boston TDWI in mid-May. A portion of the talk is basic technology theory, designed to help you convince managers that open source in the BI market is inevitable. The rest is a fast run-through of open source projects at the different architectural layers and then information about adoption and risk. 120 slides in 120 minutes, which means I'll be moving fast to cover everything.

Sadly, no demos in the session. They had to be cut to make the two hour deadline.

Is This The Terrible Secret of Space?

2007-04-07T21:27:00.000-07:00

My first reaction to the scary Hillary picture was that this is the terrible secret of space. If you have no idea what I'm talking about, well, that's pop culture for you.

Graphs as Entertainment

2007-04-05T19:16:00.000-07:00

Love this video making the rounds of housing prices as a roller coaster. You really notice the big rises and falls in the market when you experience them "firsthand".

More (Better) Information on the BOBJ-INFA Suit

2007-04-02T17:05:00.000-07:00

I'll still read the court documents, but I'm finding that Vincent McBurney read it all ahead of me, commented, and linked to some of the underlying information. No need for me to repeat any of that. You can read about it in chronological order in post 1, post 2, post 3 and post 4.

I'm not sure I agree that this is the ETL "trial of the century" since I'm thinking the focus is narrower and the patent in its broadest interpretation is not enforceable. If he's right then more lawsuits will likely follow since you could read it as a patent for "modular code" and sue the crap out of everyone, rather than more narrow and specific implementation.

Patent and copyright lawsuits are the deep-vein thrombosis of the software industry. Sooner or later, one of these is going to be big enough that it causes a tech market stroke. It's bad enough that patent searches and IP lawyers are part of starting a tech company, without worrying about overly-broad patents stifling entire industries.

Informatica Wins Round 1 in Patent Lawsuit Against Business Objects

2007-04-02T16:37:00.000-07:00

The latest case in the ETL space is an old lawsuit between Informatica and Acta that carried forward when Business Objects bought Acta. It's hard to tell if this is a slow news day or is actually meaningful to the ETL business. My guess: slow news day.

INFA went for the expected "we want an injunction to stop them from selling the product" and BOBJ responded with the usual "we'll swap out the code if the court upholds this." Depending on breadth of the infringement, that could be tough.

I read the two patents in question - 6014670 and 6339775 - and they're pretty dry reading. It's hard to work out the exact breadth of the patents since companies try to word patents to cover the maximum breadth, sort of like stretching the last of the butter on too much toast. In this case the patent could be read specifically, but certain parts are an attempt to go wide, e.g. take data from point A, transform it, and drop it in point B, using metadata and an intermediate language. If you read it loosely, you get the impression that SQL and scripting languages infringe.

I haven't seen the court documents so I can't see how and where the patent applicability was narrowed by Business Objects in this case, or what specific features were determined to be infringing. More dry reading awaits.

I'm not a fan of software patents. Most of the time they're complete crap based on so much prior art that the person submitting should be put in thumbscrews. My first scan of the patents took me down this path. An actual reading made me think that there was more substance, but I was so sleepy by the end that I might have been dreaming. Until I see more documentation it's hard to say whether this has any import for any other ETL vendors out there. At the moment, it seems unlikely given that the lawsuit is over four years old.

Another One Bites the Dust

2007-03-26T20:45:00.000-07:00

InfoWorld, one of the first IT trade magazines I started getting way back before Metcalfe was predicting giant Internet collapses , is folding up shop, another victim of the migration of this sort of material online. Really, what use is a weekly news magazine that contains 30% press releases when you can get those same releases online the day they come out, and your tech news as it happens? I stopped getting InfoWorld several years ago when I realized that it and 17 other magazines got read 1 issue out of every 4 and I'd have fifty pounds of recycling every month. I now get one trade and even that may go away.

The most-read elements of these magazines were the occasional feature and the columnists. Most columnists have their own site and make their own ad revenues, albeit probably not the same level of money unless they're really popular.

The only thing I'll miss is that there's a level of editorial control and breadth in trades that doesn't come across from RSS feeds and blogs. I expect someone will figure out the right way to pay reporters and provide industry-related journalistic content online. In the meantime, I'll surf and randomly pick from my del.ico.us links and RSS feeds to keep the diversity of input up. Otherwise I'll end up like people who read Cat Fancy and nothing else. Catminster anyone?

When to Consolidate and When to Federate

2007-03-18T19:11:00.000-07:00

The will be on data consolidation versus virtualization and the fact that we're starting to need both in the data warehouse (hence the "hybrid data warehouse" in the title).

On-demand access to current data or operational details requires some rethinking of infrastructure. If we want to physically consolidate the data then we need to rethink the extract mechanisms, schemas, storage models and even delivery interfaces to client applications.

The assumption is no longer true that data should always be stored in a central warehouse database. We have technologies that open up access to whole new realms of data: spreadsheets, files, web pages, messages, XML, images, standard industry formats. Centralizing this content isn’t required to manage the schema and relationships for these things – often it can’t.

Data federation / EII tools provide an option that lets us centralize most of the data, but provide access to information that can't easily be consolidated. Unlike direct access to the native sources, these tools allow us to mediate access and provide a single point of management and reuse.

The webcast will go into some more detail on methods of consolidation and where federated access makes sense. Wednesday at 9:00 AM, Pacific.

Ze Frank's Last Show Makes People Sad

2007-03-18T18:59:00.000-07:00

It's ironic that a show designed to make people (generally) happy can make them feel sad. I'll miss The Show. Final episode yesterday.

Fire Arts Festival in July (and other events worth going to)

2007-03-09T11:21:00.000-08:00

The Fire Arts Festival (*fire*, not fine) looks like a pyro's dream conference. I may be in the bay area at the right time, in which case I'm going to try to wrangle a ticket to the event. The pre-conference classes are as interesting as the festival, covering everything from electomechanics to concrete sculpture. Hot damn!

While I'm on the topic of events, South by Southwest (SXSW) is getting started. Another place I wish I could be instead of where I am now. My particular interest is SXSW Interactive - with sessions like "the real story behind snakes on a plane" and "AJAX or Flash: What's Right for You?" there's something for everyone.

TED Conference Happening This Week

2007-03-08T19:33:00.000-08:00

A conference you'll likely never attend, TED is a great event. The conference is invite only and it's not cheap at $4000, but they've been kind enough to make audio and video of past events available. Their blog has current news about this year's event. I had the opportunity to go to TED years ago through the fortune of being with someone who knew how to crash exclusive events and it was a great time.

The speakers are uniformly excellent, the ideas and concepts are often thought-provoking, and the people who attend are as interesting as the speakers. If you ever have the chance to attend, go. If not, enjoy the online material. I listen to these during commutes and travels, much like I listen to IT Conversations, another favorite.

Great Resource for Giving Better Presentations

2007-03-08T18:44:00.000-08:00

One of the things I keep an eye out for is good information on speaking and presentations. Today I discovered Presentation Zen, which has a lot of philosophical elements I agree with and practical advice as well. If you want to learn how to improve your presenting skills, this is a worthwhile resource.

Brilliant Boston Followup to Aquateen

2007-03-01T21:29:00.000-08:00

First they shut down the city in response to Lite Brites. Then they blow up a traffic counter. When will this Bostonian witch-hunt carnage end? I'm really looking forward to the May TDWI conference in Boston. Maybe I'll bring a karaoke machine for them to explode.

Oracle-Hyperion Deal, What Happens Next?

2007-03-01T19:15:00.000-08:00

Oracle buying Hyperion has sure set the BI industry chattering, mostly because nobody predicted it. It's fairly obvious why they bought Hyperion. What's not clear is what happens next.

Oracle was buying several things:

Best-of-breed financial and planning applications
A solutions-oriented sales force with access to CFOs and business managers
More revenue growth and market share.

This was about applications and minimally about BI, but the reality is that it's all about Oracle growing revenues and customer base through acquisition. The question is whether Oracle can successfully acquire companies the way Cisco managed to during their heyday. Based on the track record with other applications and the delays in product integration, it's a wait-and-see question.

What happens next is the more interesting question for all those Hyperion customers. In the short term, the Oracle fog of uncertainty will descend, resulting in the usual deal-killing slowdown that will hit Hyperion sales while Oracle sorts out the acquisition. There's a significant overlap in products, though Oracle was dismissive of some of their existing products in the press conference.

Expect the financial and planning applications to take the forefront, which means that overlapping Oracle products will be likely pushed into the background. It also means that customers of these applications (and the product engineers) should be safe, as should the sales force given their skill selling BI and planning on the business side of the corporate house.

The status of Essbase is more questionable. Oracle buried Express into the database, so it may be that we'll see Essbase dropped eventually, with features needed by the Hyperion applications migrated into the Oracle OLAP product lineup. Or it may stay where it is. I've seen arguments going both ways but my money is on an eventual end of life.

There's overlap in other areas, like master data management. Unclear whether the overlapping Hyperion or Oracle MDM products will survive. I would bet that reporting tools (mainly Brio) are dead and in the not-too-distant future. It's a no-brainer given all the other BI tools Oracle has.

Once they rationalize business practices we should seem some changes in Hyperion pricing. They're more expensive than Oracle, which might incent Oracle to keep the prices the same. I suspect they will migrate to the model Oracle follows with other product and surprise with (minimally) lower prices.

What happens to Hyperion employees is a different question. Oracle says they'll make a performance management group, and that means the Hyperion sales force stays intact. Product management / engineering employees will have a tougher time. There's a difference in culture between Hyperion and Oracle that could make sticking with Oracle an unpleasant option for a lot of these people. I haven't kept close watch on the Sunopsis employees, but I know of a few who left after that acquisition.

Overall, expect more acquisitions this year in the BI market. SAP still needs to work on this product area. Master data management is lacking at both Cognos and Business Objects and there are plenty of smaller vendors still active.

Full Day Conference Session on Open Source BI and DW

2007-02-28T22:46:00.000-08:00

Open source has little visibility in the mainstream data warehousing community. This is an attempt to change that. We've got four projects involved in the open source business intelligence and data warehouse course at the May TDWI. I'm finalizing the structure and lineup for the afternoon session,. The plan is to do demo-based comparison of the various projects, let them show interesting things they can do, present some short case studies, and do a panel on open source adoption.

The projects who have given me a verbal "yes" are:

The morning session will cover OSS possibilities for the entire data warehouse technology stack. The afternoon is dedicated to BI. This should be a great time and will be a lot different from my usual full-day sessions. I'll be writing more about each project as I work on open source tool evaluations for an upcoming report.

Post-TDWI Technology Highlights and Trends

2007-02-28T17:48:00.000-08:00

I always come back from a TDWI conference with a big stack of notes from meetings with vendors and people doing work in BI and performance management. This conference was larger than usual with a lot of interesting things going. Some highlights:

Rapid Increases in Data Volumes - Appliances and More Appliances
Scalability and performance are a big issue and getting bigger. I spoke to several people working on dealing with increasing data volumes. I used to plan about 20-25% annual growth in data when I managed BI and DW applications. Based on this small sample, the amount of data and requests for new information are pushing this to the 30-40% annual growth range for existing warehouses.

The high end just keeps getting higher too. This probably caught people off guard who were expecting the ceiling on data to stay relatively steady. It isn't, so speicalized products to deal with vast amounts of data are still useful.

On that front, there were interesting announcements at the show. DATAllegro had an interesting announcement, in that they're moving away from a hardware appliance to a software-only solution, but decided to partner with specific vendors. This throws more fuel on the "commodity hardware keeps getting better, why go with specialty hardware?" line of questions aimed at appliance vendors.

Nettezza, the most well-known data warehouse appliance vendor, continues to plug away, and they are still solidly in the custom-engineered hardware space. DATAllegro seems to be going down a path similar to that of Greenplum, who position themselves as a less expensive version of Teradata, with Greenplum preferably running on Sun hardware (and the hardware they prefer is pretty nice). Not a peep out of the startup Calpont for close to two years now, though I hear there may be signs of life coming from them again. I think the much older Kognitio is still breathing, though seen in the US about as often as the ivory-billed woodpecker (you might remember their offering better as White Cross).

One interesting company getting ready to launch is Dataupia, who have a very different take on how to deal with large-scale data storage issues. I'd love to talk more but I've been NDAd. I'd like to write more about Paraccel too since they position themselves as a transparent query accellerator. NDAd there as well but I liked what I heard because they tackle one of the problems faced by products based on similar architectures.

So does Hyperroll but via a very different approach. The column-based database Vertica was not in sight, a different approach to scaling query performance and data volumes. Sybase IQ is another example. At this point I'm moving away from dealing with large data volumes and more into query performance whether or not it's on large data, which is a different problem.

Predictive Analytics
Predictive analytics (aka data mining) was another hot topic, rated by the Executive Summit attendees as the number one item expected to have the most impact over the next several years. I see this as big too. A lot of the engineering problems we faced in the 90's have been addressed and raw computing power on the cheap has made broader use feasible (these technologies tend to be CPU and memory intensive).

I'm not completely sold on predicitive analytics for end users yet. It stills takes expertise to understand which techniques work best for what types of problems. Using them requires knowledge of the technology so you avoid making common data mining mistakes. Tool support is better, but still has a ways to go before it can get out of the heavy-duty analyst / IT side.

That said, I've been seeing more PA technologies embedded into applications. My opinion has been that PA is really a back-end technology almost like ETL. The output of the tool is the meaningful element; the processing isn't so interesting to users. You can see evidence of embedded analytics all over the place.

Recommendation engines are probably the most obvious of the buried uses. I think the market is ignoring recommendation engines at the moment, and there are few standalone products out there that aren't part of some e-commerce solution. When you think about the deluge of information and the growing data on the web, recommendation engines make a lot of sense. Companies like Amazon pioneeered this in the e-commerce space, but it's expanded into many other areas.

There are startups all over the place that are really nothing more than specialized recommendations engines disguised as services for consumers. Pandora and Last.fm in the music space, for example. A common element in companies from Amazon to Netflix to Pandora or Last.fm is that the recommendation engines are (largely) custom. The tooling for broad use isnt' there yet but the market is going to demand it. I'm seeing more and more in the buzz about recommendations in the web market (where I spend about half my time). I've got plenty more on recommendation engines that I've been putting into a presentation for a conference, so I'll save the topic for another time.

Bottom line: predictive analytics, coming thing, may be heading into overhype mode if the industry analysts and vendors start pushing it as a solid end-user technology.

Clickstream Coming Back?
I had a half-dozen conversations about clickstream data, up from the 1 I usually have. I haven't done a lot of work with web analytics over the past few years because the bottom more or less dropped out of that market. Most people bought web analysis packages or used hosted services.

This is changing, which I think reflects a maturing of company BI efforts. They need to marry the clickstream and internal data to get a view across processes and artificual business divisions. All the companies I talked to were talking about bringing the data in-house, even if they were planning to conitnue with the online analytics provided by their applications. The problem is simplay that web site data alone isn't as valuable as web site data integrated with back office systems.

Note to companies providing product information online: if you have a product data sheet online, I'm looking at it to see if you fit what I'm looking for. Sticking this behind a registration page just annoys me and I'll look elsewhere because you're interrupting my flow. My experience is that about 30% of the email addresses are no good, or are throwaways. It's fun to look through reg-page data for the number of entries from "Heywood Jablowme" at "123 Fake Street". Not that I would enter in that data. I use BugMeNot.

Plenty of other interesting trends continue as well:

BI pushing into the mid-market and smaller companies
BI use increasing in the mainstream majority
commoditization of BI tools (and to some extent integration tools) accelerating
on-demand / near-real-time / right-time data needs creating new architectural challenges for IT and vendors alike

Strangely missing was the impact of the web on the BI market and vice versa. The topic is largely ignored until you mention something like mashups, which are really one-off specialized BI applications built with web technology. It's like the movie "When Worlds Collide" Everything is going to be shaken up in the BI and web technology markets as web tooling, BI infrastructure, and the mix of data and text converge. At the moment we're living in parallel worlds and neither world recognizes the other exists or is headed into the same space.

The vendors I saw at TDWI that seem to be thinking about this (either deeply or opportunistically) were Cognos (Celequest really), IBM (deep down in research-land, e.g. ManyEyes and QEDWiki) and oddly enough the EII vendor Denodo.

I've been spending an inordinate amount of time on all things web and BI since I worked on both sides of the aisle. It's fun. I can't wait for MashupCamp this summer. In the meantime, I've got preparations for a web integration talk I'm doing at the Shared Insights Portals conference.

See a trend or something I missed at TDWI? Leave a comment below.

A Simple Description of Net Neutrality and Why It's Important

2007-02-27T18:50:00.000-08:00

Net neutrality is simple: we all use the same pipes. The telcos can't mess with what's inside. Everybody gets the same speed and quality of service. To quote from the Daily Show, "It's as though the richer companies get no advantage at all."

There's no development that's so successful that people driven by greed will not try to screw it up. Telcos and their partners pushing pay-for-preference schemes is like reverting to the old days of AT&T.

Save The Internet has a great short video about what's happening and you can help. They stopped one corporate power grab last year and now they're on the offensive, trying to preserve a key design principle that's made the web possible.

Head of Appropriations Committee Doesn't Believe Copernicus or Galileo

2007-02-16T10:32:00.000-08:00

One of the most powerful politicians in congress (Warren Chisum, R Texas) doesn't believe the earth revolves around the sun. Reversing 500 years of science is an amazing feat. I found this so hard to believe I had to go hunting to make sure it wasn't a hoax. Turns out it's true. I wonder what he thinks of the space program? Obviously the moon landings and mars missions were faked. We all saw how realistic Total Recall was so they must have used the same sets as the mars rover.

"Still, it's enough to set the world a-spinning that the chairman of the House Appropriations Committee, the most powerful committee in the House, distributed to legislators a memo pitching crazed wingers who believe the earth stands still -- doesn't spin on its axis or revolve around the Sun -- that Copernicus was part of a Jewish conspiracy to undermine the Old Testament."

With this sort of monumental stupidity in our elected representatives, it's no wonder the US educational system is falling apart and science research is blossoming overseas.
Here's the blog post I first read and his link to Chisum's memo promoting the nutjobs who say the earth stands still using god-designed magnetic levitation (it's right on the home page).

Lessig on Innovation and the Read-Write Web

2007-02-15T23:49:00.000-08:00

If you think web 2.0, YouTube, public policy, innovation and anime have nothing to do with each other, think again. If you think law and public policy are dull, he may change your mind. I stumbled on this lecture by Lawrence Lessig on the topics of innovation and how we got what we have with the read-write web. It's a great talk that could be sort of summed up as web 1.0 vs. web 2.0, consumer versus creator, and consume versus consume-create-share (and recreate). And how web 3.0 could end up being web 0.5 if we aren't careful.

Funny note: Lessig used the example of John Philips Sousa trashing the infernal music machine, an example I've used in open source talks I've given. His version is better because he concludes that Sousa was right :-) At least now the infernal machines can give us back what they took away a hundred years ago. Of course Lessig is more pessimistic. His official title for the talk is "The Withering of the Net: How DC Pathologies are Undermining the Growth and Wealth of the Net." and his view is that the last 4 years have eroded the economic and innovation progress in this country.

There are a bunch of other Lessig lectures on Google video. I like his performance in this one.

Information Integration Slides Are Posted

2007-02-14T11:06:00.000-08:00

I just wrapped up the enterprise information integration (EII) webcast for TDWI so here are the slides. The general goal was to explain a little about what modern EII tools are like and scenarios where they are a good fit. EII went through a small hype cycle in 2004-2005 and kind of fell flat, but the technology has matured enough to fill gaps that ETL and EAI tools aren't well suited to. The slides are available as a PDF from Third Nature.

Next month I'll be doing a webcast on a similar topic, hybrid data warehouse architecture and options for providing access to on-demand/"right time" data. This is more focused on physical versus virtual data consolidation and when to use virtual techniques.

Update: The webcast is available in streaming format from TDWI (they require registration to view).

Aqua Teen Hunger Force Filmed Planning Boston Strike

2007-02-13T11:24:00.000-08:00

"We will disrupt their workday with a mildly offensive blinking light!" Enough said. Go watch it.

I've avoided the topic of the idiotic Boston police response to advertisements for Aqua Teen Hunger Force made from Hasbro's Lite Brite toys, mistaking them for bombs and shutting down the city for a day. Honestly, if the anti-terrorist squads can't tell the difference between a flashing, blinking, attention-getting toy and a bomb we're wasting our money. Ironic that this paranoia should strike the home state of the witch trials.
Via Boing Boing

BI Vendor Evaluation Criteria

2007-02-12T12:46:00.000-08:00

I've noticed an uptick in hits on this blog from searches looking for BI vendor evaluations and criteria. While I don't publish much on this topic, an excellent resource is Cindi Howson's BI Scorecard where she sells research reports and subscriptions that evaluate all the major business intelligence vendors. If you want specific evaluation criteria, she has a blank BI scorecard that she uses to evaluate the products available from her free resources page.

I'm currently working on an open source BI tool evaluation report and will be using Cindi's evaluation criteria with extensions specific to open source. The report will be available through BI Scorecard in late spring of this year.

Data Federation and EII are Underutilized Integration Tools

2007-02-11T22:30:00.000-08:00

I'm working my way through the current EII and data federation products as part of research I'm doing for on-demand/real-time data warehousing. The new features in these products - ease of use, non-relational sources, ERP connectors, performance / scaling enhancements, multiple input and output protocols - really make them worth a second look. It's underutilized in current environments that have to deal with on-demand or "right time" data access and multi-format data.

This coming Wednesday (February 14) I'll be doing a short webcast for TDWI on EII covering a little bit of how EII and federation tools work and how they can be used both in and outside the BI environment.

In the past I haven't been a big fan of EII for a few good (I think) reasons:

EII/federation vendors are wedged between big ETL vendors on one side and big EAI vendors on the other side (long term product viability).
One class of products tended to focus on XML-only interchange, and generally sucked for relational data, particularly in the areas of performance, scalability and ease-of-use.
The other class of products focused on distributed queries but didn't work well for anything else, and were mostly limited to relational output.
The IT usage scenarios were/are not quite mature enough to provide a solid market.
Too expensive relative to the value based on use cases.

I now find myself liking the latest releases of the products. The vendors who survived the mini-hype wave circa 2005 have largely dealt with points 2, 3 and 4 but points 1 and 5 are still true.

Long term viability of standalone vendors is still an issue, though I expect the best few may survive. We're more likely to see EII/federation slip into the data integration stacks or platforms of major vendors and squeezing the smaller niche products out. We're seeing some of that happen already.

The better products still cost too much unless you have the perfect use case. The ROI simply isn't there for real-time data delivery when companies already have alternatives that meet "good enough" criteria; technologies like replication, EAI/queuing software, and CDC/ETL combinations. But...

At the same time, I'm seeing more use cases, and more urgency around some of the existing uses. For a time dashboard hype was lifting EII vendors, but the value/cost ratio of the dashboard use case was poor. Many EII vendors are now chasing the SOA market because EII shows more value there than as an adjunct to support dashboards. Try using EAI tools to make distributed multi-format queries some time. It's not fun.

I'll be writing up more as I run products through their paces. Vendors I'll be doing in-depth work with include IBM, Composite, BEA, Business Objects and several others. Look here over the coming months for more detailed technical writeups. I'm going to be tying portals and mashup servers into the mix so this will be fun.

The Periodic Table of Visualization

2007-02-07T17:57:00.000-08:00

Exactly what it says, this is a periodic table of visualization methods. Before this I never would have known what a Nassi-Shneiderman diagram was. It will be nice when Visual Literacy gets their online tutorials up. As to the periodic table, I like it!

RIP Intelligent Enterprise

2007-02-02T22:45:00.000-08:00

The paper edition, anyway. The magazine will live on in online form. Sadly, CMP laid off Dave Stodder, who's been writing, editing, speaking and publishing in the technology realm for years. He was my first editor back before IE was formed from the merger of DBMS and Database Programming & Design. I still miss Database Programming and Design. It was the only magazine on database topics that ever catered to heavily technical developers and DBAs.

DM Review is the last magazine standing that covers the whole data warehousing and business intelligence spectrum. They've been getting decidedly thinner, so it's probably a matter of time before they shift to online-only distribution. These Internets can be hell on a business model.

Good Infographic Design on the Cost of War

2007-02-02T17:18:00.000-08:00

I like the design of this graphic from the New York Times comparing the ginormous annual cost of the Iraq war to what we could have spent the money on instead. Pretty amazing what the War That Would Pay For Itself has cost us. (click thumbnail for a larger image).

I also found out that all the libraries in my county will be closing due to lack of funds. Seems the way to pay for the cost of bombs is with our children's education. (although the direct cause of this cut is because of a sleazy federal budgeting rule where clearcutting all our public forests gives us library, police and school money; however, funding to cover the cuts would be possible, were it not for the giant sucking sound that is the Iraq war)

Attention and Amusing Ourselves to Death

2007-02-02T12:01:00.000-08:00

I was listening to a talk on continuous partial attention (CPA)and it's goodness/badness and it made me want to re-read Neil Postman's "Amusing Ourselves to Death" because it's a parallel discourse written on a similar topic before the web existed. Postman asserts that different communications mediums shape communications and public discourse. A big part of his assertion is that print carries rational argument effectively but television (and to some extent radio) hampers or removes it. I wonder what he'd say to CPA and the interactive written word?

Forward from the book:

What Orwell feared were those who would ban books. What Huxley feared was that there would be no reason to ban a book, for there would be no one who wanted to read one. Orwell feared those who would deprive us of information. Huxley feared those who would give us so much that we would be reduced to passivity and egoism. Orwell feared that the truth would be concealed from us. Huxley feared the truth would be drowned in a sea of irrelevance. Orwell feared we would become a captive culture. Huxley feared we would become a trivial culture, preoccupied with some equivalent of the feelies, the orgy porgy, and the centrifugal bumblepuppy. As Huxley remarked in Brave New World Revisited, the civil libertarians and rationalists who are ever on the alert to oppose tyranny "failed to take into account man's almost infinite appetite for distractions". In 1984, Huxley added, people are controlled by inflicting pain. In Brave New World, they are controlled by inflicting pleasure. In short, Orwell feared that what we hate will ruin us. Huxley feared that what we love will ruin us.
This book is about the possibility that Huxley, not Orwell, was right.

Wikipedia entry on the book.

Postscript: Neil would say that it's a "faustian bargain." Here's an interview with Neil Postman I found on YouTube discussing the Internets, learning and information incoherence. Highly recommended.

Game Design and System Design: Get Your Hands Dirty

2007-02-01T11:50:00.000-08:00

The final conclusion of this post on game design is that "you have to get your hands dirty", which applies equally to anyone calling themselves an architect or designer in the IT world. It's one of the reasons that so many big IT projects fail; there's typically some conceptual architect/design person or team who passes off the work to the next phase and moves on to the next $300 per hour engagement, only coming back and offer advice or tweak the design a few months later but with zero learning in between. The game design post takes a few paragraphs to get up to speed but I think he makes some excellent points.

EII as a Filesystem

2007-01-31T13:04:00.000-08:00

I've been digging through a lot of technical details on enterprise information integration (EII) products and technologies lately as I prep for some webcasts on the topic. I also spend a fair amount of time on web integration (accessing web-based data, whether in pages, files or APIs) and I turned up Amit Singh's demo of MacFUSE (embedded below).

What I found novel about it is that FUSE and in particular this demo of MacFUse is the exposing of heterogeneous data as files instead of as web services, APIs or database tables. Instead we're talking about making the data (or fragments of data) available to a standard file browser. Not something I would have thought of since this isn't very "enterprisey", a characteristic of most EII tools. The procFS part of the demo below wasn't that interesting because Unix geeks have had this for years, but RSS as a filesystem and Picasa web APIs as a filesystem were both neat to see.

Which SF Writer Are You?

2007-01-31T05:00:00.000-08:00

When I was young Isaac Asimov was my idol, so imagine how happy I was that in my more optimistic moments...

I am:

Isaac Asimov

One of the most prolific writers in history, on any imaginable subject. Cared little for art but created lasting and memorable tales.

When I was a little bit older but a lot younger than now, I really liked reading William Gibson, and interestingly in my less optimistic moments...

I am:

William Gibson

The chief instigator of the "cyberpunk" wave of the 1980s, his razzle-dazzle futuristic intrigues were, for a while, the most imitated work in science fiction.

These "Which X are you?" sites are working as well as astrology and the Myers-Briggs test. Try it for yourself at Which science fiction writer are you?

Amazon S3 Not Perfect After All

2007-01-30T17:37:00.000-08:00

The Amazon storage service has gotten lots of press recently, with several IT trade rags publishing articles and case studies about using the Amazon infrastructure services. The articles have been pretty rosy, which always makes me suspicious. Everyone has problems from time to time and the talk about how customers don't need an SLA with S3 because they have 100% uptime (from quotes in the articles) is hard to believe based on the evidence.

As far as the reality of the service, all may not be so swell with S3 based on this account discussing performance issues. We're beginning to see the element missing from the trade press, namely the boundaries defining what S3 is and isn't good for.