style="margin-top:70px;" Clickstream


BI Tools for Gerrymandering

This excellent article in the New Yorker describes the current redistricting process and the gerrymandering that's taken root. What I found intriguing is the use of Maptitude software from Caliper. By feeding as much census and election data as possible into the software it is possible to point-and-click your way to a safe congressional seat in the next election.
With a few clicks, Persily changed the map from one that showed party registration in each census block to one that revealed voting results in each block. The colors ranged from dark red, for heavily Democratic votes, to dark blue, for strongly Republican. He showed voting results in about two dozen races, from President to governor and from congressman to local offices. "The whole process has got much more sophisticated" Persily went on. "Party-registration data are not the only kind of data you want to use. You want to use real election results. That's a big change from ten years ago. We have become very good at predicting how people are going to vote. People's partisanship is at a thirty-year high. If I know you voted for Gore, I am better able to predict that you are going to vote for any given Democrat in a future election."
The use of Maptitude reminds me of hospital third-party billing software I worked on in the mid-eighties. The primary, secondary and tertiary diagnoses determine the allowable rates for insurance and medicare billing. In general, one combination provides a better total rate than the others. It was illegal for a computer system to determine the ordering, but the person doing the data entry could. So they asked us to write software that would provide every combination of diagnostic combinations and display them with the billing amounts. The user then selected the one that had the highest rate. While not illegal it definitely subverted the intent of the laws at that time.

This redistricting software does much the same thing to the political redistricting process. There was never an intent to let politicians in power redistrict to make it impossible for them to lose elections. Now that computers and good BI software make it as simple to geryymander as clicking a mouse, we need laws that explicitly forbid this practice. The problem is that two thirds of politicians in office hold safe seats because of the subverted process, so it's unlikely that any reform will take place.

I'm also embarrassed to note that my own alma mater (CMU) played a part in the Pennsylvania redistricting that's now in court.

MySQL Gotchas

I've only worked with MySQL a little bit, so I'm not yet comfortable using it for data warehousing. I'll be researching Open Source databases in a lot more detail this spring in advance of a few conferences, with the goal of pumping some large volumes of data into them to see how they handle the usual range of business intelligence tasks. Expect a spate of magazine articles come mid-summer on the results, assuming my hardware is set up in time.

While catching up on some reading I came across this useful page about some of the more annoying features of MySQL. The author doesn't treat these as bugs since they are mostly documented and designed-in behavior. For example:
If no DEFAULT value is specified for a column, MySQL automatically assigns one, as follows. If the column may take NULL as a value, the default value is NULL. If the column is declared as NOT NULL, the default value depends on the column type: (...)
I did not realize that MySQL's treatment of NULL values was as strange as the examples he walks through. Worth a read if you are starting to do any development work with MySQL (not just DW experimentation) and you are already familiar with commercial and standards-compliant databases.

Excel for Visually Immersive BI

[Pacelman]No BI tool is complete until it can handle sophisticated graphical applications. I've only recently learned that Excel makes the grade, thanks to this nice demo of a sophisticated GUI(291K).
Learning how to do this makes spreadsheets much more fun, sorta like when Visicalc was first released.
[Note: there has been some trouble downloading the file from it's original source so it's mirrored here just in case]


Data warehousing, business intelligence, IT strategy and architecture, and occasional interesting bits.

Subscribe to XML feed

Bio / About Me

Check out my book

Clickstream data warehousing book cover Buy clickstream data warehousing from

Search this site or  the web

Site search   Web search
powered by FreeFind
Popular Posts
Primate programming.
Why development in crunch mode doesn't work.
Enterprise data modeling sucks big rocks.
XP Exaggerated.
Ping-pong in the matrix.
Time management for anarchists.
Is Ab Initio worth evaluating?
Job posting: omniscient architect.
Why hiring more sales people won't grow revenues faster.
Some resources for Open Source CMS.

Reading List
The Cruise of the Snark
Blue Latitudes
Everyone in Silico
The Klamath Knot
Swarm Intelligence (Bonabeau)
A three year backlog of F&SF

Listening List
Toots and the Maytals
The Buena Vista Social Club
American Idiot

Watching List
Winged Migration Quicktime trailer
Ghengis Blues
Howl's Moving Castls
A Bronx Tale

Daily KOS
Due Diligence
Boing Boing
Kevin Kelly (Recomendo)
Not Geniuses
3 Quarks Daily

War in Context
Valmiki's Ramayana
Choose the Blue
Third Nature
Mark Madsen
The Data Warehouse Institute
James Howard Kunstler
Clickstream Data Warehousing
Technorati Profile

04/01/2003 - 05/01/2003 05/01/2003 - 06/01/2003 06/01/2003 - 07/01/2003 07/01/2003 - 08/01/2003 08/01/2003 - 09/01/2003 09/01/2003 - 10/01/2003 10/01/2003 - 11/01/2003 11/01/2003 - 12/01/2003 12/01/2003 - 01/01/2004 05/01/2004 - 06/01/2004 06/01/2004 - 07/01/2004 07/01/2004 - 08/01/2004 08/01/2004 - 09/01/2004 09/01/2004 - 10/01/2004 10/01/2004 - 11/01/2004 11/01/2004 - 12/01/2004 12/01/2004 - 01/01/2005 01/01/2005 - 02/01/2005 02/01/2005 - 03/01/2005 03/01/2005 - 04/01/2005 05/01/2005 - 06/01/2005 06/01/2005 - 07/01/2005 07/01/2005 - 08/01/2005 08/01/2005 - 09/01/2005 09/01/2005 - 10/01/2005 10/01/2005 - 11/01/2005 11/01/2005 - 12/01/2005 12/01/2005 - 01/01/2006 01/01/2006 - 02/01/2006 03/01/2006 - 04/01/2006 05/01/2006 - 06/01/2006 06/01/2006 - 07/01/2006 07/01/2006 - 08/01/2006 08/01/2006 - 09/01/2006 09/01/2006 - 10/01/2006 10/01/2006 - 11/01/2006 01/01/2007 - 02/01/2007 02/01/2007 - 03/01/2007 03/01/2007 - 04/01/2007 04/01/2007 - 05/01/2007 05/01/2007 - 06/01/2007 06/01/2007 - 07/01/2007 07/01/2007 - 08/01/2007 08/01/2007 - 09/01/2007 09/01/2007 - 10/01/2007 10/01/2007 - 11/01/2007 11/01/2007 - 12/01/2007 12/01/2007 - 01/01/2008 01/01/2008 - 02/01/2008 02/01/2008 - 03/01/2008 03/01/2008 - 04/01/2008 08/01/2008 - 09/01/2008 06/01/2009 - 07/01/2009 08/01/2009 - 09/01/2009 10/01/2009 - 11/01/2009 01/01/2010 - 02/01/2010 09/01/2011 - 10/01/2011 04/01/2013 - 05/01/2013

Powered by Blogger.

Creative Commons License
This work is licensed under this Creative Commons License except where indicated.