skip to Main Content

Following Up on the Industrial-Strength Reality of our Favorite Toolset

8 Days is Average Time it Take for IT to Add a Column to a Report

A GNET Slide That I Plan to Steal.

Last week’s post about Power Pivot jobs at GNET really drew a lot of interest.  I was thrilled to see so many resumes come in.  A number of people were/are even willing to relocate across the country!  We swamped “poor” Neelesh, heh heh.  Good stuff folks.

But I also got a lot of questions – about their methodology, how to prepare oneself to play the “BIM” role in that system, how to get your own organization to see the promise of this approach, etc.

So I thought I’d interview Neelesh for the blog.  He accepted, and here we are Smile

Bill Hader wore a PowerPivotPro visor in "Hot Rod???"  SWEET!

Confirmation like this almost makes me jealous

True story:  In college, I (Rob) majored in Computer Science, Math, and Philosophy.  Trust me, that sounds more impressive than it was – I “gamed” the university system quite well and took a lighter course load overall than any of my friends.  (I graduated Magna Cum Lazy.)

So I wrote a lot of Philosophy papers while my more responsible peers were learning to program in C++.  Which was fine, because I was a lot better at Philosophy papers than I was at programming.

In a few of those papers, I thought I “invented” a new concept that was groundbreaking.  Something that would shake the world of Philosophy to its very core.  Old men in tweed suits were going to carry me around on their shoulders chanting my name.  But then, back in reality, I discovered, a few weeks later, that some well-known philosopher had written about precisely that same thing decades ago.  Happened more than once.

That burned me up.  I chuckle, in hindsight, about how angry it made me to be “scooped” like that.  One time, a wise professor advised me to view moments like that as “confirmation” rather than invalidation of my desire to be first.  I laughed him off, thought he was patronizing me.

During the Q&A with Neelesh below, I realized I might be “cured” of that naïve problem.  Sometimes it was like my own words were coming out of his mouth.  But now it was exciting, thrilling.  My wise old professor had been serious, and correct.  It IS confirmation!  And it feels good.  I was just too young and brash to understand that back in the day.

MUCH more to it than confirmation though…

On multiple occasions during the phone interview I found myself yelling “THAT’S AWESOME!!!”  See if you can spot those places…

imageROB:  You’re a “traditional” BI pro who has very much “seen the light” in my opinion.  But that transformation couldn’t have happened all at once.  When did you first start thinking that Power Pivot was going to transform the industry?

NEELESH:  Yeah, definitely not all at once.  I remember first seeing Donald Farmer’s videos about “Gemini” in August of 2009.  And I was puzzled about the strategy – here was Analysis Services modeling being performed in Excel!  Did Microsoft expect all of us BI pros to stop using Visual Studio and start using Excel instead?  It just seemed weird.

But then you spoke at our user group in February 2010 and made it clear that Power Pivot was aimed at the Excel pro.  That made a lot more sense to me, and it got me thinking.

Like all BI firms, at GNET we had been preaching the IT-driven BI methodology for years.  “Go talk to the business, collect requirements, and then build the DW first. After a few weeks if not months, deliver the very first report.” 

This a real challenge, because not only are you asking business to wait more but there is an false assumption that ALL requirements are well defined to build a perfect data warehouse. We BI professionals know all too well, that it takes a lot of time and money to build out the “bottom up Data warehouse, first!” approach, and the idea that requirements will match data realities is a fallacy!”

ROB:  “Fallacy.”  That’s a great word.  What kind of fallacy?

NEELESH:  There are three realities of data that “bottleneck” every traditional BI project.  One is that data is never actually stored in the manner that everyone thinks it is, so there’s always a lot more re-organizing and re-shaping required than what anyone expects.

The second is that everyone assumes the business is static when in reality it is ALWAYS changing.  The requirements you lay down today are always outdated, to varying degrees, by the time the project is “complete” some number of months later.  It’s hard to hit a moving target with a glacier.

The third is probably even more problematic – people are just inherently bad at understanding and communicating what it is that they actually need.  The first time they start seeing reports, they start asking all kinds of questions and requesting multiple changes, even though you’ve built for them precisely what they specified. 

ROB:  That third one has been a key point for me too.  I call it the “dark matter” of BI projects.  So the problems with BI were known to you in early 2010, and now that you understood Power Pivot was aimed at the Excel pro…

NEELESH:  I started to realize that Power Pivot could potentially address those problems.  It didn’t take us long to verify that it does, once we started applying it.  The key point isn’t even about technology really, it’s about people.

The whole point of BI is to change behaviors – stop doing things that hurt the bottom line, and start doing new things that help.  So in order to be effective, BI has to reach everyone whose behavior impacts the business.


BI Obviously Must Inform the Business.
But it Fails to Do That Unless the “The Biz” Informs BI!

“Reach” is a two-way street though.  It’s not just publishing to every important corner of the business, but factoring in their thinking as well.  The “front line” is where everything happens, and out of necessity, there’s a lot of business expertise on the front lines that IT cannot possibly assimilate.  That expertise needs to drive the BI system even more than vice versa.

ROB:  I agree 100%.  That is the magic of Power Pivot that I see with my clients when their BIM/Excel Pros start leveraging Power Pivot, but we tend to be hired directly by the business rather than IT.  So I’m still curious how a firm like GNET does it, since I expect IT is usually the one hiring you.

NEELESH:  We sit down with an expert from the business, with Power Pivot open, and just start trying to answer some of their questions.  In this step we’re building reports right there in the room with them, during the first meeting!

Of course we immediately start running into those places where the data isn’t stored in a way that supports answering their questions.  That’s good news though, since in the “old” way, those gaps sometimes wouldn’t be visible until months of ETL/DW work had been poured in concrete!

imageROB:  We’ve made it a long way without hearing the term “Data Warehouse.”  Are DW’s no longer needed?

NEELESH:  At the end of the day, the DW is still absolutely needed.  It must be built.  But we definitely no longer build it first.

When we hold show up to those first “working” meetings with the business, we usually already have built an ODS – Operational Data Store.  It’s really just a “one stop shop” of existing data – a single SQL database that’s an amalgamation of multiple different data sources from around the company.

We make zero transformations when we’re loading the ODS.  We just pump the data in, unmodified from its original source.  The idea is just to have everything at our fingertips for the working Power Pivot sessions.

As we iterate with the business – either within a single session or across sessions – we THEN rapidly make changes in the ODS to support our Power Pivot modeling.  This evolution eventually forms the design for what the “real” DW should look like when we do build it.

ROB:  That sounds precisely like what we’ve been doing with databases from the beginning.  Of course, we didn’t have a DW “pedigree.”  We just started using databases in concert with Power Pivot, in a purely pragmatic, “address whatever needs have emerged today in our modeling efforts” manner.  I knew we weren’t doing “real” DW work but didn’t know what to call it.  I started calling them “Data Marts” but I suppose now I would consider calling them ODS’s.  Cool.

NEELESH:  In our experience, 90% of organizations heavily use ODS’s, even if they already have a DW.  The ODS arises naturally in response to limitations and failures of their traditionally-designed DW.  A lot of times, people start thinking the ODS is the DW, which remains hidden behind the scenes!  So a supposedly “legitimate” BI solution driven by a DW devolves into an ODS with hundreds of one-off reports built against it.

So the ODS practice is widespread.  It’s just a natural human response to the problems.  In many ways you are lucky you approached this fresh, with Power Pivot, and never went through the old way.

ROB:  I knew it!  But I did suffer the old way once, with the football project.  That experience, and the contrast vs. the ODS / Power Pivot approach when I re-built it, really made me comfortable with our “no formal DW” approach.  I never doubted its legitimacy as a problem solving approach.  But…  confirmation still feels good.

…Part Two of this interview is now live here.

You can find Neelesh Raheja on Twitter, on LinkedIn, and at various RC Airstrips in the Twin Cities Area.

Rob Collie

Rob Collie

One of the original engineering leaders behind Power BI and Power Pivot during his 14-year career at Microsoft, Rob Collie founded a consulting company in 2013 that is 100% devoted to “the new way forward” made possible by Power BI and its related technologies. Since 2013, PowerPivotPro has rapidly grown to become the leading firm in the industry, pioneering an agile, results-first methodology never before seen in the Business Intelligence space. A sought-after public speaker and author of the #1-selling Power BI book, Rob and his team would like to help you revolutionize your business and your career.

This Post Has 8 Comments
  1. I’ve been struggling with IT for over a year. The director of development has a certain amount of DW background and he wants to create a perfect DW with 100% felsh out of metadata before making the data from about four systems available to me. Meanwhile, I’ve created a number of ODS’s just so I can get some work done and not just sit on my hands. Terribly frustrating, but I have been hostage to the time constraints of the dev group. They don’t want to “waste” their time making a bunch of one-off ODS’s, but that leaves the rest of the business wasting resources by making guesses instead of informed decisions. Perhaps this series will give me some ammo. :-/

  2. Excellent interview, and I agree completely. I have been doing this approach at my company this year, to help drive out the design for the underlying data marts. This helps the business gain great confidence that they are going to get what they need. And it helps IT know that whatever data structures it builds in the data mart, will be used by the business.

  3. Hi Colin

    I hear you and feel your pain. Not knowing sufficient details, here are some generic pointers that may help you tackle IT objections. May be you have already tried some of these….
    1. ODSs are critical to long term success of BI as ODS abstract DW from source system changes – ERP replacement, acquisitions (new systems), upgrade etc. Highly recommended to build ODS to feed DW tables. don’t consider ODSs to be one off.
    2. Single ODS is always better and its best to have one DB with all tables but a different schema for each source system.
    3. Don’t position PowerPivot as an alternative to DW. It is an enabler for better DW design. Use it to gather and/or explain requirements. Also help IT see the queries needed to source data in to DW and realize a better design of the model as needed by business.
    4. Last but the most important one, build a PowerPivot model with the data you have, share it with other/more business folks. be sure to explain/demo how easy it was to build and don’t forget to mention “you only wish if IT would leverage this tool it to accelerate the DW build process.” Yeah…I call it peaceful negotiations!

    All the best and watch out for the next post!

  4. The biggest attention grabber here: “people are just inherently bad at understanding and communicating what it is that they actually need.” This is probably one of the most underrated but important skill sets I look for when hiring.

  5. I also find myself using the “poured in concrete” phrase when describing what I make here in the IT department. To often I get an incomplete request but am expected to read the person’s mind and magically create what they want before they even know what they want.

    Then, once I create something using an IT tool that is “poured in concrete”, I inevitably have to get out a jackhammer to break up the concrete in order to handle the changes that the user wants.

    I suppose surgery is another good analogy. An IT person wants to know what they’re operating on before they go in for surgery so they don’t have to keep opening the report up, then sewing it back together again.

    We began using Power Pivot at my company earlier this year and I have noticed that I am now getting more intelligent Data Warehouse requests from the Power Pivot users. They are able to save me a lot of time. I definitely see Power Pivot not as an enemy of the Data Warehouse, but as a good friend.

  6. I’m a VBA developer — the bridge between IT and Management. I have to tell you, that slide about “8 Days to add a column to a Report” is the reason I’ve been able to support my family for the last decade. So true.

Leave a Reply

Your email address will not be published. Required fields are marked *