A New York Times article from a few months ago really startled me.  The headline was “U.S. to Be World’s Top Oil Producer in 5 Years, Report Says.” This contradicted everything I had known about the slowdown in American oil production.  So what was the game-changer?

There are several components of the sudden shift in the world’s energy supply, but the prime mover is a resurgence of oil and gas production in the United States, particularly the unlocking of new reserves of oil and gas found in shale rock. The widespread adoption of techniques like hydraulic fracturing and horizontal drilling has made those reserves much more accessible.

Until recently, my concept of oil drilling was like this diagram:oilwell.jpg

Find a pocket of oil buried underground, put an oil well on top of it, drill until you hit that oil pocket, and pump until the well runs dry. But most of the untapped oil in the world is not in nice neat reservoirs. It is trapped inside rock formations that are spread over great distances horizontally. To unlock this reserve, two key technologies are required:

  • hydraulic fracturing or “fracking”: creating fractures in rocks and injecting fluids to force the cracks open and release trapped oil and gas
  • horizontal drilling: the ability to drill horizontally, thus be able to follow the natural direction of the oil and gas deposits


What does this have to do with QlikView? Well, we don’t know who first coined the phrase “Data is the new oil” but it is clear that organizations of all kinds are scrambling to unlock the value of their data. Unfortunately, they are still using old technology that can only drill down into data and unlock small bits of value at a time. This is not only true of traditional BI techniques of creating data cubes (thus limiting users to summary views of the data and preventing them from finding the insights hidden in the details) and predefining drill paths (thus limiting users to a narrow line of thought through a report). This is also true of many of the new generation BI tools that promise “ad hoc query” and “multidimensional drill paths” with snazzy visualizations to boot, but hide the fact that underneath the glamour is the same old SQL query on a single data source.

You may already know about QlikView’s distinctive associative experience, which shows with every click what data across the entire data model is associated and what is not. However, what is often less well known among those looking for a truly intuitive and agile BI platform is the fact that QlikView allows users to have that experience across different datasets simultaneously.

A brilliant example of this was recently shared by a customer speaker at a recent QlikView Technology Summit. Larry Griffiths, BI Manager at Bentley Systems, Inc., works at a software company with a broad portfolio of products catering to the infrastructure engineering industry. Their primary data source was SAP BW. With their existing BI tool, they struggled to enable simple tasks like pipeline and contract reviews for sales managers. They also had other data sources, such as log data that tracked actual usage of their numerous software products. Trying to analyze all that information together was difficult.  After spending a year and half searching for a solution, they selected QlikView.  In the words of the speaker, the breakthrough came when with QlikView, “…within half a day, on a little 4GB VMWare instance, we had pulled in 200 million rows of contract data and usage data.  A problem we haven’t been able to solve for a long time was solved.”

After they deployed QlikView, users from across the company found many ways to extract real value from data, via the ability of QlikView to drill horizontally and ‘frack’ the data for its valuable insights.  For example, the product development group made use of an affinity market basket analysis app, which answers questions such as “what products are used most with what other products?” By doing so, they saved over $2.5 million in software development costs in the last year by dropping unnecessary software integration projects.

They are experiencing amazing discoveries by tying multiple data sources together. The latest effort is combining revenue data from SAP BW with usage data from the application server with training data from their learning management system (LMS). What insights could they extract with this? For one, they will be able to correlate the training level of their user base with higher usage of their software and increased revenue. This gives them actionable insight across the entire company to improve customer training programs, upsell training seats, and increase revenue via better bundling of relevant software and training.

To visualize this with the drilling analogy, other BI tools can only deliver this siloed view of data, which not only keeps data fragmented, but leaves users frustrated and resorting to using Excel to get at the insights they need, which defeats the purpose of the investment in BI. QlikView gives them a holistic view, which enables them to extract maximum value from data.  Isn’t that what we all want?


Click here to watch a recorded webinar by QlikView and Bentley Systems.


Do you have a story of “horizontal drilling” with QlikView?  Please share it below!

In one of Charles Dickens’ novels, a young English orphan boy named Pip received a large sum of money from an unknown benefactor and was told he would go to London and learn how to become a “gentleman.”  Charles Dickens entitled his novel “Great Expectations”, describing how Pip felt on his way to the metropolis of London. In classic Dickens’ style, things are never what they seem and Pip’s fortune does not lead to a life of comfort and ease.Pip-magwitch.jpg


The theme of Great Expectations came up in a number of different ways on my recent trip to the metropolis of New York City to attend and present at the Big Data Summit organized by CDM Media. The presenters and delegates were a ‘who’s who’ assembly of people whose titles begin with the letter C: CIOs, CTOs, CDOs (Chief Data Officer), and even a CAO (Chief Analytics Officer), from some very well-known enterprises and brands, including American Express, Citi, AIG, Allstate, Suncor, and the National Basketball Association.


I certainly had great expectations going to the event: surely the industry luminaries would have Big Data all figured out and I would be able to come away with a better understanding of Big Data use cases.  The first speaker heightened my expectations – he gave jaw-dropping statistics about how Big Data helps the healthcare industry fight the annual loss of two hundred billion dollars to fraud and how Big Data helps a wind energy company optimally place its wind turbines by crunching massive amounts of weather data.


Another speaker gave an impassioned call to train our schoolchildren in technology and mathematics so they could become data scientists, helping corporations and nations gain a competitive edge.


Three surprising commonalities came out of these presentations and nearly 20 private conversations with these executives:

  • They don’t have Big Data all figured out. Successful projects with large ROI are few and far in between, many are still in the experimental phase.
  • Adoption is low. One executive said, “My problem is getting my people to actually use the expensive data warehouse we bought.” His data warehouse had over 125 terabytes of data.
  • Innovation comes from data mashups. The executives were most excited about the possibilities when internal data is combined with customer data, sensor data, and news feeds to deliver new services that don’t currently exist.


What did exceed my expectations was the high level of engagement the executives had as they learned about QlikView in relation to Big Data. The top reasons were:

  • QlikView helps them walk before they run. When asked if they are making the most use of their existing “small data,” not one person said yes. They saw how QlikView’s Business Discovery platform helps them achieve immediate return on the data they have, rather than waiting years for their Big Data projects to complete.
  • QlikView helps them explore their Big Data. Because creating data mashups is so intuitive and simple in QlikView, they can take extracts from their Big Data source, mash them up with other data sources, and explore the results in real time without the usual query lag and laborious data modeling work.
  • QlikView reduces dependence on data scientists. Everyone who saw a demo of QlikView for the first time was impressed with how intuitive it was to answer question after question simply by clicking, without having to create more visualizations or hire data scientists to do so. Since QlikView can scale from “small data” to Big Data, it uniquely addressed their short and long term analytic needs.


Like a great novel with unexpected plot twists, I entered the conference with great expectations, saw them brought low, but then ended with a fresh excitement over what QlikView can do for the largest of enterprises wrestling with the challenges of getting broad value from both small and big data.

For more information, check out QlikView’s page on what Big Data can do for you.

Who doesn’t like pizza? In every office where I’ve worked, free pizza is a great motivator. From a cost-benefit perspective, free pizza every day might be an effective employee incentive!

What does pizza have to do with Big Data? A recent white paper published by CITO Research and sponsored by QlikTech showcases many transformative uses of Big Data by businesses, including a large pizza chain.pizza.jpg

The pizza chain, like most retailers, needed to understand what products customers were buying and optimize their product mix to maximize profitability. Menu items that were unprofitable needed to be eliminated to make room for profitable items (which hopefully included one of my favorite pizzas - the Hawaiian, which is topped with ham and pineapple).

They faced the same challenges many retailers face when it comes to analyzing data – a complex organization comprising the corporate entity and franchisees, and the need to present the data appropriately, whether to the company board or a franchise manager.

With QlikView, the pizza chain was able to accomplish some great feats:

  • Analyze 57 million transactions
  • Consolidate data from 35 separate sources
  • Cover 500 pizza sales outlets across multiple venue types
  • Allow franchisees to view relevant datasets in a central location over a secure browser interface

What I found most impressive about the story is that this QlikView app took two developers only ten working days to complete.

With other BI platforms out there, after ten days, the developers would probably be still designing the requirements for a data warehouse just to consolidate all the data. The project would have taken months or years to complete and cost the company hundreds of free pizzas to motivate the developers. 

I think fundamentally this is why BI developers are so passionate about QlikView. They can deliver incredibly useful apps within days to business users. Not only does that make them the hero, it frees them up to tackle other interesting business problems.

The moral of the story? With QlikView, companies could offer free pizzas to their employees and still be profitable!

When I’m asked what customers use QlikView for, I usually respond with the top of mind answers about dynamic reporting, actionable dashboards, and works-the-way-your-mind-works type of analytics. There’s nothing wrong with those answers – that’s why most customers buy business intelligence software, and QlikView in particular. Last week I attended the first-ever QlikView Technology Summit in the San Francisco bay area where I got to mingle with several hundred QlikView customers and prospects. One customer’s use for QlikView blew me away.


This customer works for a healthcare company. He told me their nurses use QlikView. Not too surprising, I thought to myself, as many hospitals use QlikView to analyze patient data. He then told me how QlikView helps nurses diagnose patients. As patients describe their symptoms, the nurses make selections in the QlikView app. QlikView displays a list of possible diagnoses, narrowing down the options as the nurses make more selections.

Wow! This is the power of QlikView’s associative experience at work. If you’ve ever been to a specialist at a hospital, you usually fill out a very long questionnaire about your symptoms. This annoys me because most of the questions have nothing to do with why I’m there. By using QlikView, the nurse is able to quickly zero in on the problem because all the unrelated options are greyed out as soon as the nurse makes a selection. You might notice that this creative use of QlikView, an ostensibly BI tool, has nothing to do with charts, visualizations, and dashboards. And unlike so-called “expert systems,” where nurses follow a predefined path of questions much like a traditional call center, QlikView allows nurses to follow the patient’s own train of thought and their own experience of their condition.

I can immediately imagine a list of tangible benefits:

  • Faster time to diagnosis
  • More accurate diagnosis (because a newly-trained nurse has access to the same expertise as a seasoned pro)
  • Better patient experience (because they aren’t being asked lots of unrelated questions over and over)
  • More efficient and lower-cost healthcare

At a time when the quality and cost of healthcare seems to be at odds with each other, QlikView shows how hospitals can deliver better care at a lower cost.  Now that’s what I call a win-win situation!

Don’t Suffer in Silence

Posted by Chester Liu Jan 29, 2013

People who suffer sometimes think they are the only ones, and that no one else understands their pain and suffering. They suffer in silence. That is one of the reasons why support groups exist for various conditions and chronic illnesses. Being part of a support group of people dealing with similar issues breaks that misconception and points the way to solutions that, when shared with the group, lead to genuine life changes.


Likewise, we as knowledge workers in business tend to think we are the only ones who are not Excel jockeys or have trouble making sense of the data we have to work with. We get frustrated with our data and the tools we are given. As a result, sometimes we end up making decisions based more on instinct than on facts.

On a recent webcast for CIO magazine, I co-presented with Jeff Brobst, VP Financial Planning & Analysis at McAfee. (You can watch the recorded webinar in its entirety here.) Jeff mentioned a number of pain points that existed within the finance group in his organization that many of us can easily relate to. The finance team traditionally spent three weeks at the end of each quarter on the closing process.

Their existing BI tool was available only to a limited group and it was very difficult to combine and analyze data from heterogeneous data sets. Most of the team’s processes were manual, tedious, and time-consuming. The result was more time spent gathering and formatting data and less time on real analysis. Furthermore, multiple sources of truth and inconsistent use of data led to more work and fact-checking.

Jeff showed how QlikView changed all that. His team was able to consolidate data from multiple sources and create useful interactive dashboards that contain up-to-date data throughout the quarter, not just at the end, making it possible to course correct more than just four times a year. The closing process went from three weeks to three days. Broad access to near-real-time sales data enabled McAfee to sell products to customers at the right time at the right price.

At the close of the webcast, an attendee asked me a great question: “How would you compare the ROI of QlikView versus other products?” My answer was two-fold. On the investment side, QlikView is a bargain when you consider the months or years it takes to get a traditional BI stack deployed compared to the weeks or even days it takes to get QlikView deployed. The savings alone in man-hours is huge. On the return side, customers who switched to QlikView often find that BI becomes not the domain of the few but rather the essential, go-to data analysis tool across the company. This democratization of data analysis pays handsome dividends because now all decision makers are using the same data and asking and answering their own questions in order to improve business.

At McAfee, QlikView is truly transforming the culture in terms of how data gets used and produces value. Jeff Brobst commented that those who see QlikView naturally want access to it, and adoption is growing rapidly.

You don’t need to suffer alone. It’s time to join the support group for people who are frustrated with their current data analysis and reporting tools—it’s called QlikView!

Big Data Is for Everyone

Posted by Chester Liu Jan 24, 2013

All of us have probably experienced this in our childhood: Off in the distance, we see some of our classmates playing a game we have never seen before. They are grouped in a tight circle and we can’t quite see what they are doing. Out of curiosity we walk closer, but we are not sure whether it’s something we should participate in. Maybe it’s boring or a waste of time. On the other hand, if we don’t find out what they are doing, we might be missing out on something really fun or important. At last, we gather up the courage and ask that all-important question: “Mind if I join in?”


That is how many business people feel about Big Data. On the one hand it’s a buzzword rippling through industry like a secret whispered around a classroom. On the other hand, it might be just a distraction from getting work done. The predicament business people find themselves in is how to determine whether they ought to be asking, “Mind if I join in?”

CITO Research has just published a QlikView-sponsored whitepaper that seeks to address this. It introduces Big Data in an easy to understand format and shows the relevance of Big Data to business challenges. It also compares and contrasts various approaches to getting value from Big Data. Finally, it concludes with two very engaging stories of how Big Data, when put in the hands of everyday users, has been able to achieve truly transformational outcomes for their organizations.

We hope this whitepaper helps you with your understanding of Big Data. You can download it here:


My previous blog about the human mind and Big Data (see “Your Split Brain (Part 2 in the ‘Body of Big Data’ Series)”) ended with the realization that Big Data has two fundamentally different use cases. If managers are to get first-hand value out of Big Data, it might be useful to understand how the conscious, or sensory-somatic, part of our brain makes decisions.


Photo by: Johns Hopkins University Applied Physics Laboratory

If you pick up something with your hand (try it right now – pick up an object in front of you) you’ll notice that you first use your fingertips to touch the object, your curl your fingers to grasp the object, you move your hand to orient the object, and then you move your arm to raise the object. Of the millions of sensory receptors from your fingertips to your arm, you actually need very few of them at any one time to perform this action. Consider people with prosthetic limbs which enable them to do the same thing you just did. This proves that it doesn’t take much data to perform an action. However, out of the millions of receptors, you never know which ones you will need for a particular decision. For example, if a mosquito bites your arm, a very small number of receptors fire furiously, causing you to swat the mosquito.

This brings us to my second insight about making decisions when we have Big Data: Use data relevant to the current problem.

While the algorithms data scientists write may need most of the data most of the time, managers almost always need just a relatively small amount of relevant information to make their decisions. This means that whichever BI tool you use to analyze data must be able to present information that’s relevant to your particular decision at hand. Just as it doesn’t make sense to kill a mosquito with a sledgehammer (and you’ll probably miss), it doesn’t make sense for managers to use the same tools as data scientists to make business decisions.

What are the best ways to make sure we have relevant information at our fingertips, then? Here are a few ideas:

  • Aggregate data to the level of the decision to be made. For example, it doesn’t make sense to look at time-series data at a one-second granularity level when you’re making a decision that is at the granularity of a daily level.
  • Organize your decisions into categories based on the task or app concept. Just as picking up objects and swatting mosquitoes are two entirely different decisions requiring different data, consider creating apps with data models that contain just the relevant data. Don’t throw everything you could possibly need into one monolithic app or dashboard; you’ll drown in the data.

Hopefully these insights have been helpful.  In my next blog post I’ll explore the mystery of how our minds make decisions based on both conscious and subconscious information.

No, I’m not referring to the left and right lobes of your brain or implying that you have a personality disorder. In my previous blog post, “Who, Me? I’m Big Data? (Part 1 in the ‘Body of Big Data’ Series)”, I described how our nervous system is like Big Data, dealing with volume, variety, and velocity continuously.

nervous system.jpgMy further research into how our nervous system works revealed that there are two primary subsystems: the autonomic and the sensory-somatic. Before I lose you with the big words, just remember that:

  • “Autonomic” is automatic. It’s about things you don’t control that happen automatically, such as your heart rate, the size of your iris, your digestive process. You might notice them when they aren’t working properly, but most of the time they are complex background processes that optimize themselves to your body’s current condition. For example, when you are exercising your heart rate goes up and when you are resting your heart rate goes down. Somehow your autonomic nervous system takes into account all that Big Data delivered by your senses and optimizes your body’s functions.
  • “Sensory” means you feel it and “somatic” means you’re awake. So the sensory-somatic nervous system represents things you are conscious of such as sights, sounds, and movement. In this realm of the mind we make conscious decisions based on sensory input. When you hear the roar of a lion you start to run, but when you see that the lion is in a cage you decide to stay put.

So how does the way our mind process Big Data relate to how organizations process Big Data? The corporate analog to the autonomic nervous system is any use of Big Data that is automated, repetitive, and usually involved with process optimization. For example, online retailers use Big Data algorithms to optimize their recommendation engines to drive incremental sales, or hedge funds use automated stock trading algorithms to optimize their return on assets. These are uses of Big Data that are the domain of data scientists – people who are skilled in mathematics, statistics, and algorithms and create batch processes that run in the background continuously.

The corporate analog of the sensory-somatic nervous system is the managers who make conscious decisions informed by data. The decisions they make each day are different and cannot be described programmatically. One question often leads to another, and one decision is usually part of a series of decisions.

This thinking led me to an insight:  Big Data has two fundamentally different use cases. What works for the data scientist is probably going to be different from what works for management. In my next blog post in this series, I will explore the thought process of managers more deeply, to understand what is important to aid them in better decision-making.

When it comes to the topic of Big Data I have to make a public admission. I have a split personality. On the one hand the geek in me, from years spent as a software engineer, relishes the challenge of installing my own Hadoop cluster, writing MapReduce algorithms, and running all sorts of performance tests to see for myself how amazing the technology is. On the other hand, as a pragmatic product marketing manager (yes, I did take the class), I just want to get stuff done and understand my data ASAP, without writing a single line of code.


A series of deep questions dawned on me. Why do I think the way I do? Why do I sometimes prefer one thing over another? What drives me to make certain decisions and not others? The quest for these answers led me to a surprising connection between our human bodies and Big Data. This connection led to five key insights which I will share through a series of blog posts.

I first searched the web for some interesting statistics on the human nervous system. If Big Data is described by volume, variety, and velocity, how does that relate to the human body? For example, our eyes have 1.2 million fibers in our optic nerve. Given that our eyes can detect flickering at around 30 times per second and we have two eyes, I’m making a rough estimate that our eyes transmit over 60 million signals per second. Our nose has 12 million receptor cells. Receptor cells fire at various intervals, usually faster when there is a new stimulus and slowing down under repeated stimulus. That’s why when you walk into a coffee shop you are at first intoxicated by the wonderful aroma but after you’ve been inside for a while you don’t notice it anymore. So between our other sensory body parts such as the tongue, ear, and the largest of all, the skin, our brain is receiving at least hundreds of millions of signals per second – covering both variety and velocity.

How about volume? The newest figures suggest we have around 86 million neurons in our brain which record a massive amount of sights, sounds, sensations, etc. Each neuron is equivalent to much more than a transistor on a CPU because each one is connected to ten thousand other neurons, on average. That means we have roughly one quadrillion (1015) connections in our brain. So our brain is basically an amazing massively parallel and scalable processing device with co-located storage – which sounds like many Big Data architectures, doesn’t it?

Where am I going with this? You are Big Data! To better understand how organizations can get value out of Big Data, it might be useful to study the human brain in a little bit more detail because it has been fine-tuned over millions of years. In my next blog post, I will delve deeper into how our brain uses all this Big Data.

If I started talking to people about the credit card “platform” I bet I would get a few strange looks. We take it for granted that we can pay for souvenirs with the same credit card in San Francisco as in Singapore. What makes something a “platform” is its ability to openly interconnect with other systems to accomplish much more than what it could do by itself. For example, the credit card platform ties together merchants, card issuers, banks, payment processors, and consumers into one seamless shopping experience.

credit card.png

In the same vein QlikView is a platform. The best way to imagine it is by describing the opposite: a closed system. Imagine a BI product that limits you to only the data sources for which the vendor distributes connectors. Imagine if the visuals you could create are limited to only the ones that come out of the box. Imagine if the visuals you create are stuck in your BI application and can’t be embedded in another application or web site. That would be a very limiting experience indeed. In today’s heterogeneous business environment, no vendor can possibly anticipate every customer’s data or visualization needs.

QlikView is already a vibrant BI platform with a growing ecosystem of more than 1,400 partners. We recently launched QlikMarket, an online solution exchange where QlikView customers can find QlikTech-vetted connectors, extensions, and apps. Connectors provide pre-built connectivity to business applications such as SAP, Salesforce.com, and PeopleSoft, plus social media sites such as Facebook and Twitter. And that’s not to mention the new visualization extensions and apps that are appearing every day.

With “QlikView.next” (the code name for the next generation of the QlikView Business Discovery platform) we’re taking the platform concept to the next level, under the Premier Platform theme. We are investing in QlikView as the heart of an ecosystem, facilitated by QlikMarket.


The four scenarios we are addressing within the “Premier Platform” theme are:

  • No Data, No Discoveries. The ability to access, transform, and manage data is prerequisite to enabling free data exploration and discovery of insights.  QlikView will provide developers with a modern browser-based development environment where drag-and-drop is the norm. Through a library of scriptlets and expressions, your organization can standardize definitions of KPIs and other important calculations.
  • No More Reporting. QlikView is about action- and activity-based decision making. QlikView will allow users to print exactly what they see and tell the data story via an interactive storyboard paradigm. Those who still need to create static reports for audits and record keeping will be able to export data from QlikView to third-party reporting engines.
  • Simplified Partner Deployments. This scenario is aimed at making it even easier for our OEM (original equipment manufacturer) partners to add QlikView analytic capabilities to their products. QlikView.next will include APIs (application programming interfaces) that permit access to QlikView data and visualizations by a middleware server. QlikTech OEM partners will be able to create sophisticated QlikView extensions, use QlikView to drive an associative experience, and create sophisticated mashups that use some of if not the entire QlikView user interface.
  • Unified Platform Interface. Last but not least, this scenario means that QlikView will be exposing the exact API that the native chart objects use to communicate with the QlikView engine. That means customers and partners will be able to create any visualization they can dream of and integrate them with QlikView to the same degree as native QlikView charts.

QlikView is poised to become a BI developer’s dream platform. Imagine integrating with any data source or destination, creating any visualization, and doing it all within an ultra-modern interface. I’m jazzed to see this dream take shape.

To learn more about “QlikView.next,” the code name for the next generation of the QlikView Business Discovery platform, download the white paper, The “QlikView.next” Product Scenarios.

Filter Blog

By date:
By tag: