Where next for Hunchworks?

Hunchworks is just one of the technologies that we designed this year.  It’s time to talk about why, and what some of the others are.

Why build Hunchworks?

Imagine the scene today. I’m an analyst at the UN, and most mornings I get to my desk in the morning and think “what’s going on in my area” or “something’s not right here”. What do I do about that today, and what would be useful to me next year if I want to do it better?

Today, I rely mostly on personal connections, colleagues, skype chats, newsgroups and websites. If I want to know what’s going on, I log into accounts in several different systems (my UN and personal emails, a group of Skype chats, some irc channels and a set of newsfeed websites) and scan the entries in them for things (specific event types, geographical areas etc) that are urgent, or interesting, or that I might be able to help with. If there’s something important, I then go to a new set of sites and start digging around in data and messaging experts I trust to verify what I’ve seen and what I think might be happening, and then start thinking about what I can do to help with the situation and who I need to start telling about it (and how).

This is wildly inefficient. It contains time lags at every step, it has me searching through time-ordered items and repeating thoughts and work that other people are probably doing at their own desks too. And I miss things: I miss emails in the mass of group mailings, I miss announcements and important data hidden in the mass of writing in every channel when an event’s going on, and half the time if I don’t deliberately save something when I see it go past I’m unlikely to be able to find it again.

Next year could be different. I would like to get to my desk and log onto one system to see my alerts overnight (yes, I’ll still have to look at the others, but I would like to have one place to start). I’d like to see a dashboard of new hunches that something might be happening (that have been created either by humans with that “something’s not right” feeling or data sniffers checking through the webosphere for first signals) in areas that interest me, a list of things that I and people connected to me are already working on. If I have that “something’s not right” feeling myself, I’d like access to newsfeeds and data that I can look through for clues to why I’m having that thought, or support for a hypothesis I’m forming about it.

And so on. The bottom line is that I need access to data, tools, actions, people and their existing hypotheses about the world. And some of that’s going to be specialist. Which is why we’ve been building Hunchworks.

Where to go next

We’ve built the basic system (and you can help too), but there’s a lot of development still to do. Here’s the Hunchworks feature pick-list for next year; the one that we’ll show users and ask ‘which of these will help you most’. How much can be developed relies on Global Pulse funding, and also on how far other organisations get with similar work in the open source domain. But here’s the first list. We’re looking for external partners to help with these projects:

• Complementarity search. Allow users to search for users with skills and interests that complement their skillset or the skills needed for their hunches.
• Improved trust. New trust metrics, and improved algorithms for rating combination and interest ranking.
• Better links to data sources and conversations about them, including twitter, open-source data, city data and satellite data.
• Hunch splitting, merging and cluster algorithms, based on the Paris text-analysis algorithms developed for proof of concept study.

And we need to do this work either in-house at the UN or with close support from an in-houseteam:

• Hunchworks plugin for UNDP Teamworks. This makes Hunchworks available to the whole UN Teamworks community. Although since Teamworks is a Drupal site, it might be better for us to aim at building a custom Drupal module to handle this connection instead.
• Hunchworks LinkedIn application. Allows a user to form a hunch, then invite their LinkedIn contacts to comment on it.
• Report generation plugin. Many UN departments still have their own reporting formats and procedures. Yes, we could lobby to change that, but for the moment it’s easier to map what they are and make sure reports in the right formats go to the right people in each organization, and that Hunchworks users know what information they need to be gathering for this.
• Connection to a data science toolbag.

Other technologies are needed to support this vision. In the next few years, our analysts will need:

• Security. Reassurance that their data will not be accessible (or that it will be very difficult and take a lot of time andresources to access) by people they don’t want to access it. Also reassurance that if an incursion is attempted, it will be quickly detected and anythingconnected it isolated and checked.
• Localisation. The ability to access Hunchworks and other tools in their local language.
• Federation. The ability to access Hunchworks from local nodes in bandwidth environments ranging from completely offline to intermittent to low-band and the speeds we sometimes see in Manhattan.
• Access from multiple devices. Because when you’re out in the middle of nowhere, the only sure thing is that your Internet connection will fail. But your phone (and specifically the SMS on it) might still be okay.
• Intelligent agents. We’ve started building simple agents that monitor real-time feeds, but we need to build this concept out so the agents are actively helping the human Hunchworks users.

Work that will have to happen, will affect Global Pulse capabilities and timelines, but is outside the scope of Global Pulse includes:

• Data standards definitions.
• Data science toolbag (containing useful data gathering, cleaning, analysis and visualization tools). We’ve been talking to Data without Borders about hosting this for the world.
• Data science tools (excluding Hunchworks).

We’ll be helping with these too where we can.