Clickstream

Another Botanical Trip

I'll soon be off on another botanical expedition, this time to the eastern Siskiyous. Someone recently asked me "why botany?" The answer is simple. Native plant ecology (and ecosystem studies in general) is a lot more challenging and complex than dealing with software, which makes it more rewarding. It requires more holistic thinking, deals more with emergent properties than reductionist approaches, it's important to everyone, and it allows me to work on my farmer's tan.

Why this and not something else, say molecular cell biology where I got my start? Because it's more challenging than poking around in DNA, and field biologists seem to have a lot more fun than gengineers, glow-in-the-dark mice and fish notwithstanding. It must be the fluorescent lighting that brings people down. Genentech had real nice offices, but the lighting...

Posted by Mark Thursday, June 26, 2003 12:22:00 AM | permalink | (0) comments

Critical Path Transmogrifies Email and Snarfs Your Information?

I found the information about Critical Path in this rumor report to be interesting. Maybe someone there read the book and decided that the information about URL rewriting and redirection was too useful to pass up.

Basically, every time an email recipient clicks on a URL in one of those rewritten email messages, it would generate an entry in CP's web server logs. With some simple analysis of the logs - plain vanilla Analog would do - they would have a nice report of what people are interested in linking to, what they click on, percentage of clickthrouhgs, etc. If they have enough email traffic then they could gather a lot of useful - and salable - information in a short time.

I do find it hard to believe that a service provider would reach into their customer's email messages and rewrite all the URLs without their knowledge, but worse things have been done. My guess is they are doing it and providing analysis to the people who use them as a service provider, rather than doing this on their own. I wonder if anyone has confirmation in the form of email that went through their URL transmogrifier?

Posted by Mark Wednesday, June 25, 2003 11:36:00 PM | permalink | (0) comments

Databeacon Java-based OLAP Client Tool

I occasionally will post something about a BI product without doing an in-depth description or analysis. I read through Databeacon's PR and documentation for their "Collaboration Edition" and looked at the demo and think it's worth a short note.

Without putting the tool through its paces on my own warehouse I can't say anything about the relative goodness of the product, however there are a few things I liked about this product that make it worth investigation in the future.

What mostly caught my interest was their use of a downloadable Java applet. Vendors touted this when Java was new and not really ready for it yet. This time around it looks much better. I like having a more robust client than standard HTML-and-forms interfaces normally deliver (although DHTML has made these much more palatable too). The ability to send out the results in multiple formats is a common feature in most BI tools, but the web clients of many have seriously brain-damaged limitations and crufty implementations.

My biggest complaint on this cursory first pass is their web site. Flash intros, frames and subframes that don't render properly in Mozilla all bug the heck out of me. More on the product when I have time to do a thorough evaluation.

Posted by Mark Tuesday, June 24, 2003 8:45:00 AM | permalink | (0) comments

Software Support: Should We be Paying for Bug Fixes?

The software contract review time of year is one of those periods most people dread. There are invariably changes to costs, licensing, service and support, all wrapped in byzantine terms and conditions. Software vendors in general, and BI tool vendors in particular, have never been particularly good about this process.

One fact some data warehouse managers may not know is that Microstrategy is based in one of the states that passed a variant of UCITA , self-help provisions and all. UCITA is a legal morass pushed by some of the biggest software organizations that includes numerous draconian provisions to render customers powerless. This came up during my negotiations with Microstrategy two years ago. I had to pay a law firm to review the contract and provide specific anti-UCITA clauses to incorporate into the contract they wanted us to sign.

Picking on MSI isn't my goal - the experience of dealing with them and UCITA provisions is just an example of how the legal deck is stacked against customers. My real complaint extends beyond these legal problems to the support and upgrade provisions of software licensing.

I fully support paying for software upgrades. If I want the latest features, or compatibility with the latest major release of Oracle, paying for the upgrade is reasonable. What is unreasonable is paying for features that were supposed to work when I bought the software, and paying for fixes to problems in my software. Software quality is a serious problem with hard costs to the business, both IT labor costs, user labor costs for workarounds, and the hard costs of business interruption.

Durable goods have warranties. Even my toaster comes with a warranty. If there's a major problem with my car due to a design or manufacturing defect then the manufacturer will likely fix it, even if the warranty expired. Yet I'm expected to pay 20% of the purchase cost of the software every year for simple fixes.

I propose that bug fixes, support and upgrades be unbundled. Instead, software companies provide bug (and security) fixes free of charge. Paid support should be reserved for actual support: me calling them for help in configuring or using the software. Upgrades can be combined with support, or paid for separately.

Pulling the upgrades out of the contract also allows upgrades to be purchased on business or technical merit. This has the obvious disincentive for the vendor because they now have to sell me on the the benefits of the latest version. Other disincentives include loss of revenue now that customers aren't paying to correct poor quality software, and how to handle the invariable obsolescence of supporting technology.

Maybe with this buyer's market and the building backlash against software costs and missing ROI we will see a vendor take a bold step forward. Wouldn't that be nice?

Posted by Mark Monday, June 23, 2003 9:32:00 PM | permalink | (0) comments

Surprise! Technology Innovation is not Directed at Things Which Improve Our Lives

I just read this article concerning the fact that technological innovation is mostly directed at things which will not substantially change or improve our lives (reference courtesy of /.). It's a new article on an old topic: materialism versus a higher purpose.

It shouldn't be a surprise that most technology development is directed towards less than useful or noble purposes. Development is driven by commercial interests trying to make money. Research for the public good or for intellectual pursuits is out of fashion. Government funding for basic research is down. Very few corporations do any basic science research, and most do very little applied research outside the narrows bounds of their own products. When the society at large does not invest, how can one expect "noble purpose" to be served?

Finding and serving needs that are of true value while making a profit is not easy to do. Much that is worthwhile has no direct relation to turning a profit. Until there's a societal change to value other things more than profit - something unlikely in a system that runs in a positive feedback loop of consumption, commercialism and media - the process of innovation will be directed toward less than noble ends.

Still, the article is worth reading if only to provide a pause from the breathless boosterism of the techno-industry.

Posted by Mark Wednesday, June 18, 2003 10:13:00 PM | permalink | (0) comments

My Latest Article on Real-time Data Integration is Out

I have a feature article on real-time data integration in the current issue of Intelligent Enterprise. I was originally writing a longer piece on how to select the appropriate technologies for real-time data integration but ended up publishing this condensed version.

The basic idea is that there are only a few very basic conceptual models people use when talking about data integration: point-to-point, hub-and-spoke, and bus (not the diesel kind). These models impose a certain view of the applications and how they are integration, and provide the language and ideas for convincing others that data integration is basic IT infrastructure.

I think this point is missed repeatedly. Integration is almost always viewed as a point solution between the system being built Right Now and systems that already exist. To avoid the huge maintenance trap of spaghetti code and system connections that one-off integration creates, it has to be exposed and managed as a class of infrastructure just like the networks on which it exists.

Below the conceptual model that defines the "what", there are interaction models that define the "how". How does one application communicate with another? How does data get from point A to point B? This really boils down to the mechanism: publish and subscribe, direct or brokered messaging, and so forth.

All this stuff is at the architecture level. It's abstract and the missing part is what the components of the architecture are. The article discusses them, but not in detail.

Once you have all this in your head, it's pretty easy to evaluate vendor's all-encompassing products, point solutions, and ways to roll your own, which is probably the best overall, provided you're given the time to properly design it. Sorry, but XP won't cut it when laying in infrastructure. You sometimes have to design before coding.

Posted by Mark Monday, June 16, 2003 1:28:00 PM | permalink | (0) comments

Clickstream Related News: Netscape/AOL Busted for Monitoring User Downloads

Well, busted for a meager $100,000 for violating their user's privacy. The NY attorney general's office has a press release about it, as does Wired and several others.

I wasn't aware of the Netscape's download manager tracking downloads and sending the data to AOL back when I wrote the sections on privacy and on mechanisms other than simple web server logs for data collection in the book. I thought I thoroughly checked out the ongoing practices of non-browser software like RealPlayer and various spyware like the Comet Cursor, but I missed this one. Too bad, since I was using the software at the time. It just goes to show that you can't trust anyone, particularly when that anyone is a giant media conglomerate losing money as fast as the US treasury under Bush.

Posted by Mark 12:48:00 PM | permalink | (0) comments

HP Discovers Fundamental Computing Particle While Reinventing the Service Bureau

Outsourcing is all the rage in corporate IT these days. What's odd is that my first real (corporate-like) job was in a service bureau providing data center services to hospitals. It was a lot like working for the company in the movie "Office Space", only worse. This was in the dying days of service bureaus and I left after about 9 months. That industry was a legacy of the days when computers were expensive shared resources.

We're seeing a resurrection of that industry, only now it's called outsourcing and the outsourcer provides more custom IT services than the "any color you want so long as it's 3270 green" model we used to follow. I'm glad the computer industry has a short institutional memory. That makes it much easier to recycle work as if it's new and different. Too bad patent and copyright abuse are putting an end to this. I like it when everything old is new again.

I noticed HP's latest announcement on a model for charging customers for computing resources, but not due to originality of concept. Everyone used to charge for mainframe cycles and storage. I took a college course in Service Level Agreements, chargeback models, and all sorts of stuff I though I would never use thanks to the PC revolution. IBM recently made a similar announcement.

No, HP caught my eye because Computerworld quoted them in this article and I saw originality of nomenclature:

Under HP's scheme, prices would vary based on factors such as the overall demand placed on servers, storage devices and other IT resources, said Bernardo Huberman, an HP fellow and director of the systems research center at the company's HP Labs unit.

He added that a new unit-of-computing metric, which is being called a "computon" inside HP, would be akin to the pricing models that utilities use to charge customers for kilowatt-hours of electricity based on the ebb and flow of power demand.

Computon?? Not to be confused with computron, one of the fundamental particles used in the study of quantum bogodynamics I thought it was a joke, but I've been assured they're serious.

It's too bad they need an HP fellow to work out how to handle chargeback. All they need to do is hire back some retired mainframers or buy my old textbook. I'll let it go for a reasonable price.

Posted by Mark Wednesday, June 11, 2003 12:20:00 AM | permalink | (0) comments

Floods, famines, fires - disaster stories don't get any better than this

I was at the Siskiyou Field Institute and the related Conference on Klamath-Siskiyou Ecology. It was a fun time, as it is every year. These events are often more fun than the tech conferences I normally attend or speak at, possibly excepting Etech. I believe it's the field work that makes biologists and ecologists so laid back and fun, or maybe it's facing scary and unavoidable conclusions changes one's character.

After some field courses at SFI, the three days of the conference went by really fast. I can summarize the conference succinctly:

We are doomed.

If that was too short, here are some highlights of the sessions.

A disease (phytophthera - same genus that caused the Irish potato famine) that is killing all the oaks in the bay area is spreading. Port Orford cedar disease (also a phytophthera) is spreading and kills 96%+ of host trees. These two diseases will remake most of the low elevation ecosystem on the west coast, and current forest management practices on our public lands are making the problems worse. Loss of these trees means no trees in many areas, which means an ecosystem collapse for which nobody can predict the outcome other than minimal wildlife and probably a lot of weeds. The oak phytophthera hybridizes with the one that causes chestnut blight (which destroyed the chestnut forests of the eastern US) and this hybrid kills poplars and willows. If that happens on the west coast then most of the western trees excepting conifers will be wiped out.

Combine the old phytophthera plagues of the east coast and Europe with the west coast and the spread of different P's in places like Australia where 25% of the plant cover is dying in some regions, and wildlands worldwide are not looking good. The diseases are becoming a permanent part of the landscape and transform it unpredictable ways. These diseases affect commercial food crops, which is where it will hit home for most people. The problems in food crops will increase thanks to companies like Archer Daniels Midland and Monsanto, who focus on single-gene resistance to disease in their seed crops so they can sell new varieties every few years. Biotech is not an answer. More on this topic when I write about bioinformatics and decision support another time.

Getting away from thoughts of mass extinction, there was some talk of forest management and fire planning. The US forest service's management of federal forest land is based on almost no science other than federally mandated timber quotas. Of the little science present, most is 25 years out of date. Pity the USFS biologists, who have no power when dealing with the timber managers. Dealing with ecological problems in the USFS is like asking a wheelwright to fix your car. Most of the time will be spent looking for the horse.

Fire is normal in most forests, and serves a useful function ("duh" is the proper response to this but we need scientific publications to prove it to the Dept. of the Interior). Tree plantations created by the timber companies and the USFS create a situation where fires are both more frequently and severel. Coupled with drought, the managed lands are tinder boxes, far worse than the unlogged forests. The Bush administration will protect our communities with a new fire plan that involves, ahem, cutting down the large, old fire-resistant trees in the wilderness and doing nothing at the forest periphery or in the plantation zones, near where people actually live. Meanwhile, some new research shows more roads in the wilderness increases the frequency of fires.

My travels through the Biscuit fire area (largest in a century of Oregon history) last fall and more extensively this spring show one thing for sure: our managed lands are far worse off than the unmanaged wilderness areas. Pictures are not yet up (I need about 25 MB more space than I have available). The forests are recovering nicely from this huge fire. Flowers everywhere. Trees resprouting in the most severe burn areas. Knobcone pine seedlings starting. Some of my favorite areas were burned to a crisp, to use the technical term, yet when standing in the middle of the burned areas it doesn't feel like the disaster the national press were saying it was.

Climate research based on high-mountain lake sediments shows a shifting of plant species upward, indicating a warming trend throughout the region. As usual, long term, who can say for sure? All we know is that since the 1800s it's gotten toasty mighty fast.

On the animals front: Klamath river salmon and steelhead killed by the Bush administration in the largest fish kill in US history may take decades to recover. Mammals are decreasing in population, excepting opossum and deer mice (the mice like clearcuts - too bad they harbor human-transmissible diseases). Migratory birds populations over the past 20 years are down 50% on average, region-wide. Pacific fishers and other predators, decreasing.

I came away from this conference with a sense of foreboding. It's the feeling one might expect the protagonist would feel in a Greek tragedy as the prophet's words about the inevitable begin to sink in, or the feeling a red-shirt must have when Captain Kirk asks him to join the landing party.

Lucky for me that I'm an optimist. A cynical optimist.

For those at the conference who were interested, plant taxonomy and post-fire pictures will be available in a week or so.

Posted by Mark Tuesday, June 10, 2003 12:43:00 AM | permalink | (0) comments

Soon I Will Be Rich

In the past week I received business solicitations to use my bank account for the transfer of a total of $268.3 billion, of which I get to keep about 15%. I'm amazed at how much money government officials in Gambia, Congo, the Phillipines and Nigeria have available. It just goes to show the slant of US news media, always portraying these countries as debtor nations with economic problems.

I've been suggesting some of my proposed business partners who are in the same country get together and pool their resources into my bank account because they're sending me email from the same Internet cafes in the same cities. Those banking fees can really add up.

Posted by Mark Monday, June 09, 2003 10:36:00 PM | permalink | (0) comments