Building data pipelines

How to make data the gift that keeps on giving

When I was a kid, my favorite toys were LEGOs, because I could put them together in an endless number of ways. Even so, I would always ask my parents for the latest cool electronic toy for Christmas. When I was 12 years old, I got a battery-operated toy robot I had wanted—Verbot, by Tomy. I was so excited. What fascinated me was that it had voice commands as an interface, like something straight out of Knight Rider 2000 or HAL 9000, or Joshua in Wargames. But it turned out the interface was clunky, and there wasn’t much range to what it could do. I played with it for about 10 minutes, and then I cracked open the dome and took the whole thing apart, much to the chagrin of my parents.

It wasn’t the first time, or the last. I spent a lot of time as a child disassembling things. To be fair, I also built a lot of models, but once they were built, the fun was over. They were puzzles that once completed had no utility, except to put them on a shelf and admire them. It was the LEGOs I returned to again and again.

It strikes me that this is a very good analogy for the experience that we’ve been having with data and analytics.

The word “data” comes from the Latin word “datum,” which means “something given.” Something given could also be described as a “gift.” 

So data = something given = gift. 

Every aspect of an enterprise’s value chain generates these “data gifts,” if you will.

Now, if you will continue to indulge my love of word origins . . .

The word “analytics” comes from the Latin word “analisis,” which means “a breaking up, an untying.”

If we dig a little further and look at the word “analysis,” “ana” means “throughout” and “lisis” means “a loosening.”

Analytics literally sets data loose, presenting an endless opportunity for curiosity, learning and developing insight. (Caveat: loosening does not mean letting go of security, controls, and respect for the data gifts we receive—after all, “curiosity” comes from curiosus, which means “careful” 🙂

Or at least that’s what they should do.

Too often the data gifts we get with analytics are like electronic robots. We get something that has been assembled for us, we explore it for about 10 minutes, and then we have another question and we want to be able to do something else with it. But we can’t because the data set wasn’t built to do that, any more than Verbot was built to be able to extend its arms out sideways. In both cases, to get it to do something different you would essentially have to take it apart and rebuild it.

I never had any success doing that with my toys; it was beyond my capabilities. But we do have a way to do that in the world of data analytics: we call in a specialist who does have those capabilities–an analyst or data engineer—who goes back to the data, extracts, transforms and loads it and builds a new pipeline to deliver a new data set for analysis. It takes a long time, and unfortunately, what we get is a modified robot. It might have some new or modified functionality, but essentially it is still a limited set of components that have been assembled for use in a pre-defined way. You can’t take those pieces and make anything new.

For data to be the gift that keeps on giving, analytics have to set it loose in a way that makes it more like LEGOs than a robot. What we really need is something that provides some structure and guidance, but that gives business people the flexibility to build the way they want to, in a far faster time frame, so they can enjoy the gifts of data sooner. They need to be able to take it apart and build something new, as many times as they want to. And they need to be able to add new sources of data with new attributes, almost as easily as you can add new types of LEGO bricks to your collection.

That is what Direct Data Mapping does. Once you discover the attributes of a piece of data through Direct Data Mapping, then you know what you can build with it. There’s some minimal assembly required as we acquire the data and put it on a platform, but then you essentially have a growing pile of LEGO with which you can build.

Without that capability, the anticipation we experience with enterprise data gifts quickly turns to frustration: we have a sense there’s something more, but it remains just out of reach. Time and money are wasted; potential business value is lost, and peoples’ curiosity is squandered. 

It’s the same experience I had with Verbot. As a child, I tried to satisfy my curiosity by taking a screwdriver to it. I suppose that was my way of getting more value out of that gift, though once I had it apart it was a great disappointment to discover that this thing that had captured my imagination was in fact extremely limited. For my parents, it was perhaps seen as a waste of money. 

Enterprises need the freedom and power to keep re-experiencing the gift of data. They need to stay curious in order to experience and realize business value from their data in new and meaningful ways. When it becomes like that gift that you open and just can’t stop playing with, then the words “data” and “analytics” will live up to their true meanings.