The UX of Big Data

Following on from my Dangers of Big Data talk at DunDDD, I’ve been thinking about what a good user experience for data analytics would look like, imagining the business user presented with useful, actionable information rather than notepad and a copy of the R or Python cookbook. I want something deceptively simple like the Google search box, rather than deceptively complex like Excel.

Excel, and R and Python, put a lot of tools at your disposal, and you could use any of them to construct an answer, but the secret to analytics relies on getting a valid, useful answer. The first is a matter of restricting the answer space to that which can be supported by the data (for example, disallowing multiplication of time-based input streams, or aggregating when there is no statistical basis for it), the second is a matter of allowing the user to explore the space so they can determine (and where appropriate, train the system to recognise) which factors are most important, how they affect the desired outcome, and how changes to the environment affect these factors.

Then the question becomes, how much should the software take over. Do we have a duty to protect users from themselves by preventing invalid analysis where we can detect it, or do we have to accept that the frustration that will cause leads to alienation and users will be less likely to respond well to further corrections. Even nudging had its possible, as anyone who had been frustrated by grammar checkers can attest. But at least nudging helps the user to understand, rather than putting up roadblocks. Nudging encourages learning, roadblocks encourage switching to another way.

How would you encourage users to handle analysis appropriately?