skip to Main Content

One of the Greatest Movie Ending Sequences of All Time

“I’ve…  SEEN THINGS… you wouldn’t believe.”
-Roy Batty

Do you experience “lightning bolt” moments?  Instants where something just hits you so clearly that your perspective changes permanently?

Personally, I live for those moments.  They are rare, of course, but oh so worth it.

I recently spent a couple days in San Diego, setting up a client with a self-service BI system based around PowerPivot.  I came away with several new (or newly-strengthened) observations, a couple of which were delivered in that lightning bolt manner.

Sharp People + The Right Tools = A Lot Happens in Just 3-4 Hours

In order to understand the observations, you really need to know the context of what we were doing.

The lightning bolts hit late in my second and final day of working with these folks.  We’d spent about a day and a half doing hands-on training with their data sources – I find that more effective than canned training with sample data.  Of course, to cover enough ground in a short period of time, you have to jump around a lot.

So it wasn’t until the last few hours that we settled in and focused on building a single complete model, end to end.  In those last few hours, we did ALL of the following:

  1. Sketched out the goal – a particular report/model geared at a particular business purpose.
  2. Discovered flaws in the underlying SQL sources that needed to be corrected.
  3. Grabbed the DBA, explained the problems.  He fixed them in real time and gave us a new query to use.
  4. Successfully built the report envisioned in step 1 – a report that simply DID NOT EXIST anywhere at this company beforehand.
  5. Explored the report (by now I was mostly an observer, and they were driving).
  6. Determined that the report, which they had wanted for months, was not actually the report that they needed!  (Only after seeing and playing with the data was it clear that the answers were elsewhere).
  7. Sketched out a NEW model and report based on that new wisdom, and built it
  8. Explored the report using a variety of different visualizations.
  9. Confirmed some of their long-running hypotheses, and gained some new insights.
  10. Discovered mysterious anomalies in the data, investigated them, and explained them.
  11. Identified improvements that could be made on the data COLLECTION side (in their customer-facing systems) in order to enable an entirely new class of deeper insight.

Yes, ALL of that transpired in just 3-4 hours.  Rapid Iteration indeed.  In one afternoon we did what could take a year:  2 full iterations on their core model.  Plus #11 is just crazy – feedback into the operational side of the house in order to enable v3.

My Jules Winfield “moments of clarity” follow.

“Failure” is an Inevitable Part of the BI Process… so Fail Quickly

Look back at the list above.  Steps 1-5 are a cross-section of what happens in a traditional BI project:  Sketch requirements, execute, debug, deliver.  Of course, this all happened in about 90 minutes, as opposed to a few weeks or months.  I’m starting to get used to that.

But look at step 6:  How many traditional BI projects chew up weeks/months/money, deliver 100% what you asked for, but ultimately, tell you much less than you were hoping?

To implementers and consumers of traditional BI solutions, I bet that sounds a lot more familiar than you’d like.  After all, BI is as much a human process as it is a technology process.  Here’s the crux:

Requirements definition is only informed by what you know TODAY!  And since BI projects are by definition aimed at providing information that you currently lack, BI requirements definition always involves a certain amount of guesswork – much more guesswork, in fact, than we typically acknowledge.

So BI projects sometimes dramatically under-deliver relative to expectations, and through no fault of the players involved.  Of course, that’s little consolation when tons of time and money are down the drain.  After that amount of sunk cost, it’s awfully hard to pony up the time and money for Round Two isn’t it?

But in an effective Self-Service system with PowerPivot, Round One only took 90 minutes and four people!  We weren’t exactly discouraged – in fact we were EXCITED to have learned what we really needed to build 🙂

Side Effect:  Do Your Traditional BI Prototypes with PowerPivot

This is obvious and people have been saying it for awhile now, but just in case:  even if you are planning a traditional BI solution, do yourself a favor and implement Round One in PowerPivot.  Flush out all of these initial insights up front, refine requirements, and only then commit large chunks of time and money to the formal project.

…Continued Later This Week

This post is already running a bit long, so I will save the rest for the next post.

Plus, the most controversial and unapologetic observation is yet to come, and deserves its own headline.

Rob Collie

Rob Collie

One of the original engineering leaders behind Power BI and Power Pivot during his 14-year career at Microsoft, Rob Collie founded a consulting company in 2013 that is 100% devoted to “the new way forward” made possible by Power BI and its related technologies. Since 2013, PowerPivotPro has rapidly grown to become the leading firm in the industry, pioneering an agile, results-first methodology never before seen in the Business Intelligence space. A sought-after public speaker and author of the #1-selling Power BI book, Rob and his team would like to help you revolutionize your business and your career.

This Post Has 4 Comments
  1. I’m so happy to read this story! For two reasons that I’m going to explain.
    First, I always thought that PowerPivot is the most powerful prototyping tool existing today. When you start the analysis with a customers and you enter the “BI Commando” mode (but also “BI A-Team” would be a good name), the risk of having a traditional SSAS environment is that those 30 minutes you spend building the prototype model… are just too much. Your counterpart, someone who know the wisdom, is lost looking at VS Studio that he simply don’t understand. But when you are in Excel 90% of your time, well, he feels more comfortable. He sees his own data. He understands what’s happening. He’s engaged.
    Second, you just defined a methodology of work that we might call “Agile BI” which is very useful in the data exploration phase. One or two smart guys might be enough to prototyping data modeling, ETL, data cleansing and reporting in a very fast and closed loop. Yeah, these guys have to be very smart, they have to talk the language of business with the customer and the language of SQL with the DBA, but at the end of the day they can make the difference. We might start to formalize this approach in a business proposal instead of making it happen as a random result during what was supposed to do a “training on the job” session.

    And we are probably just at the beginning… 🙂

  2. Rob: great post. My favorite take away is this bit: “Only after seeing and playing with the data was it clear that the answers were elsewhere.” You’ve identified PowerPivot as an enabler of iterative BI…for the customer. A sort of “Rapid BI development” (the only acronym I get out of that is RABID, which I would rather avoid :-)). But the idea is key. We haven’t had this agility before, and it certainly will lead to a better success ratio. That, of course, provides full support for your advice of using PowerPivot for prototyping on traditional BI projects. I have been saying for quite a while now that PowerPivot will end up being a key tool for enterprise BI pros, and not just power users. You’ve shown why that’s the case.

  3. Hey guys, thanks for the comments, sorry it took me so long to respond.

    I think we are all in agreement on PowerPivot as a superior BI prototyping tool. I really like Marco’s observation that even using the traditional tools in the presence of the customer is a “lose the audience” moment, and for BI to succeed, the audience must always be engaged.

    Going beyond the POC angle, however, I’d like to know more about what Andrew is hinting at with “Agile BI.” For instance, the clients that were the inspiration for this post – PowerPivot is going to BE their only BI system. It’s neither a supplement to existing BI tools (they have none), and it’s not a prototyping tool. It is THE tool. Anecdotally at least, I am seeing a lot of that – people viewing PowerPivot as their chance to break into BI in a manner that is both time- and cost-efficient.


Leave a Reply

Your email address will not be published. Required fields are marked *