Global groups

And so onto the global groups that I’m tracking or part of.

This is only the start of a list that I’ll add to when I remember all the job-related groups that I’m involved in.

New York tech groups

I’ve been talking to quite a few people as part of the dayjob.  And I knew I was talking to quite a few communities, but even I was surprised when our communications officer asked just how many, and we counted over 100.  Which since I’ve only been doing this since January is rather a lot.  So first, if you’re part of a community that I talk to, please bear with me if I seem a little distracted. And second, part of the job is to create stronger links between these communities and the UN, so I thought I’d write about who I was talking to and why, to give other UN peeps some possibly-needed leads.

This is going to take a while, so I’ve split it into three posts: international communities, local New York communities and UN projects and communities.  Later on, there are academics, companies and conferences, but for now the focus is on groups.

So. Starting with the New York communities (and this is a post that would be appropriate for each PulseLab).

I found these communities through personal contacts (it helped to have shared an apartment with a major community member – thank you John, and thanks to the crisismappers for introducing us), and by searching meetup.com.  Garys Guide was also useful, as was searching eventbrite and the New Work City and General Assembly events and classes pages. But after a while, most useful leads come from people I meet.

Community events happen, usually in the evenings, at tech company buildings all over the city, but there are two co-working spaces that are very active at hosting community events and well worth a mention: New Work City and General Assembly.  I’ve been a member of both of these – for a once-weekly desk space at New Work City, and for community notices and some excellent tech training classes at GA.

For algorithms, I go to:

  • Hacks/Hackers New York – hackers and journalists together: a very active group, always with interesting ideas
  • NYC Predictive Analytics – applied machine learning and big data
  • NYC data nerds. Not so much a group, as an informal data drinking club, brought together by Drew Conway and Hilary Mason. We’ve also heard about a Sunday-morning data nerds’ brunch club by the New York Times building, but we haven’t managed to track them down yet.
  • Data Without Borders – not so much a meetup as a movement that runs hackathons and projects, but a good one worth watching.

For programming, I go to:

  • NY Hacker – the local coders group. Holds monthly townhalls and weekly drop-ins
  • NY Tech – the big meetup for NYC technologists, hosts huge meetings that I haven’t managed to get to yet.
  • NY Ruby, nyc.rb and NYC on Rails – the Ruby programming language. I also like Ruby Nuby, a group that meets to train people in Ruby.
  • NY Python and Django-NYC – the Python programming language
  • NY R – the R statistical programming language (useful for big data)
  • HPC & GPU Supercomputing Group NY – these guys can run up some awesome computing resources very very quickly
  • NY Hadoop user group – one of the more widely-used big data frameworks
  • NY Scala – a newish framework that I’m tracking as potentially useful to us in the future

For development (as in human development) technology and Open Data:

  • NY Tech4Good
  • OpenNY – our local Open Data group, and a strong part of the New York open data initiatives.
  • Volunteer Coders – doing some great work on social projects.
  • CrisisCampNYC – runs a monthly social that gets the local crisismappers away from their keyboards for a while.

For hardware:

And for sanity, I go to the NY CTO Club.

I will have missed some groups off that list, some of them because they’re not quite up and running yet. It’s not that I don’t love you, it’s just that I’m tracking a lot of groups here – so please please tell me about anyone active that I’ve forgotten.

WherecampDC thoughts on place

(Apologies – I’m tidying up my backlog of draft posts, so some of the following posts will be a little late and/or sketchy.)

Most objects have a physical geography.  Some of the ones that might not (the argument continues) are part of the Internet, but most objects and the data they generate can be tied to a specific place or places (e.g. a crisismapper in London mapping Haiti on a server in Luxembourg has three).

So is what we’re doing here an extension of human geography, the study of the interactions between people and place?

Maps contain objects that are important to people. So a ‘conventional’, bought-over-the-counter paper map will contain things like roads, trees, buildings, post offices, hospitals, trig points.  I’m wondering today if we’re adding vulnerability overlays or complete layers to our maps.

For instance, if a conventional map contains health facilities and we’re tracking an epidemic, then are we creating the ‘shadows’ of those facilities.  Is deforestation the shadow of trees? Food crisis the shadow of markets, fields and roads?

It’s also interesting to think about the touch points between visualization and gis.  If a visualization has place, does that become a map?  Or a vizualisation informed by a map?

Hmm… time to wander over to the W3C Points of Interest working group, and see where they’ve got to with this.

New development data

I’ve spent almost two years now thinking about and doing humanitarian crisismapping – about the sources, analysis and communication of information available to organisations like the UN before, during and after a natural disaster (fire, earthquake, tsunami, floods, snows etc), but that’s too diverse and uncertain for a small number of people in these organisations to distill into usable, searchable knowledge in the timeframes involved in any disaster (seconds, minutes, hours, days before it’s all over and all responders can do is help people to recover from its effects).

There have been some amazing things done in those years. The UN cluster system, although imperfect and not mandatory to join, does have a structure that field organisations can join to improve their combined effort in an area (nutrition, health, water/sanitation, emergency shelter, camp coordination, protection, early recovery, logistics, telecomms). Teams like UN OCHA, the standby task force, crisiscommons, sahana, humanitarian open street map, humanity road and mapaction are building maps and situation reports from data across the internet and reported from disaster zones, have helped open up new data sources ranging from near-real-time satellite images to gazetteers and citizen journalists, and built new tools, processes and techniques to do this.

But quietly, behind the big stories like Haiti and Libya, there’s also been a lot of work on how to map and use new sources of development data. The same sources, tools and techniques can also be used to map water availability, health outbreaks, famines and migrations.

But there’s more. The crisismappers have been using people (lots of people, all over the world) to hunt through open data streams like twitter, facebook, website updates, news streams and satellite images, and creating tools to help with searching and tagging information in these (this is exactly the type of search through data that happens as part of big data analysis). But also out there are slower forms of datastream on the internet, like website contents, updates to blogs and open data from governments, companies and institutions. And data with a lower psychological distance but more uncertainty as to intent, like trends in the searches made on google. And if you’re part of an organization, you also have access to your institutional data (and incidentally should be asking yourself what the barriers are to making this data, a cleaned version of it, or results from analyzing it, openly available), and to data from other institutions and companies. You might also have access to sensor outputs and people’s knowledge and opinions.

And more. A lot of the big data systems out there are designed to infer people’s behaviours and status from the text (twitter, facebook etc) and data (phone positions etc) they output. Which is great, but a bit one-way and unfocussed. Which is why crowdsourcing systems like ushahidi and open311 become important. And just one of the reasons why the hunchworks system is being built.

Attracting Opposites

Most search applications are about similarity – about finding items or people who are similar in some way, and ranking their appearance by that similarity (amongst other things –there are also things like connection and advertising cash to take into consideration). I have a slightly different problem to solve: that of suggesting connections and forming teams between people who are complementary.

Which is a great problem to have: new turf is always exciting. But before I get too carried away, I need to look for places where this problem might already have been solved. The obvious places to look are management theory, robot team formation and autonomous agents. Each of these is focused around a goal or task that requires a set of skills, time availability, location availability etc., and includes the study of communication, coordination and cooperation between team elements. In robotics, this is autonomy theory. Multi-agent theory is also a rich ground, as is team forming (from team development models).

A lot of human team theory is about personalities and how they complement each other – the completer-finishers and plants of the Belbin Team Inventory and group behaviour. We need to be aware of these (and there is much work in community management on to manage personality conflicts), but they’re not the main focus of this problem.

A lot of search now includes the networks connected to an item – for instance, the pages cited by and citing other pages, the people linked by social networks. That gives us similarity again, which isn’t what we’re looking for, but the network idea is powerful and still potentially useful to us. As our system develops, we should see teams forming around ideas, and groups forming around subject areas. Perhaps we could mine those groups and the people profiles in them, to template the types of team that form around different problem types and topics. Combining that with a ‘skills needed’, location, tags etc template for each hunch might produce a very powerful team suggestion tool.

Which brings us to robotics. There is a lot of work (e.g. here) on team formation in multi-agent theory and autonomy. Mostly these are goal-based, an extension of things like belief-desire-intention models from single agents to multiple cooperating actors.

There are overlap areas too – like the I-MINDS system for creating student teams.

And then there’s the whole area of autonomy that’s dedicated to mixed human-robot teams. Which is exactly what we’re planning to do by creating agents (wrappers for data feeds and sniffers) as users on a par with the human users in our system.

There’s a lot of work here, something worthy of at least one thesis methinks. But we have to make it work, and quickly, so the trick for now is to think about the above, and create something simple and intuitive that ‘just works’. Onwards!

Development crises – who’s doing what?

A lot of organisations, communities and individuals are working on human development. The UNDP Sudan CRMAT team are working on mapping work in specific areas, and having been on a 3W team and browsed the Worldwide NGO directory, I’d say they have a lot of work to do. But although looking at who is working on development is interesting, we need to narrow down our search to people who search for, analyse and response to development crises.

There is a lot of information online about natural disasters and other drivers (UN Global Pulse found 39 early warning systems in the UN alone), but less about the route from effect detection to decisions to act. Warning isn’t action, and we need to feed, track and encourage this cycle rather than creating yet another information-only system.

So who’s looking? And who is able to act? The shortest answer to that is “everybody” – when we start to be individually affected by a crisis, we’re usually aware of it, and begin to act to mitigate our own exposure and risk to it (for example, the Coping Strategies Index is a way of quantifying how people behave, in general, in response to a food security problem).

Organisations charged with preventing, mitigating and responding to development crises worldwide include the UN, USAID and World Bank, amongst dozens of others. There are global monitoring systems in place, although their ability to access and use data is still limited (which is why organisations like Global Pulse were formed): most UN agencies have people working on the ground (usually in a UN agency office loosely connected to a resident coordinators office) who are at least partially aware of local crisis effects, and at least one team, HEWS, works and shares development crisis data across organisational boundaries. The HEWS website currently focuses on natural drivers of humanitarian crises, but does also contain a contingency planning toolkit that could be adapted to track longer-term issues (e.g. post-disaster effects on local agriculture).

The development world is shifting to help communities become more resilient, i.e. less likely to be severely damaged by a development crisis. The UN is working with governments on sustainable development, which “meets the needs of the present without compromising the ability of future generations to meet their own needs”. Organisations like Alnap and ICT4Dev are starting to discuss structural drivers and mitigations – there are several good points and potential mitigation strategies for drought, for example, in Alnap’s “Humanitarian action in drought-related emergencies”, and of particular interest is their comment on the parallel coping systems (zakat, migration, remittances, local NGOs, community help) that Western donors have tended not to track during responses.

Community knowledge (both receiving and improving it) is also becoming important to resilience work. The World Bank and organisations like Civic Commons are working with Open Data teams in the developing world to improve community knowledge and resilience. This is a welcome step beyond large, western, funding models but we need to think about how to connect these and grassroots, community-based detections (e.g. Ushahidi data) of crisis effects to existing top-down monitoring systems?

Development crises

I need to understand what human development is, how it can be measured, what interruptions (‘shocks’) and damage (‘reversals’) to it look like, and the methods that people are using to a) reduce the risk of development reversals or b) reduce the impact of shocks when they can’t be avoided. And then look at who is doing what and think about how we might do more to help this work.

First, human development. Wikipedia is surprisingly insightful (ed: not really – it’s been lifted from the UNDP website) about this – “Development is … about expanding the choices people have, to lead lives that they value and improving the human condition so that people will get the chance to lead full lives. And it is thus about much more than economic growth, which is only a means – if a very important one – of enlarging people’s choices”… “The most basic capabilities for human development are to lead long and healthy lives, to be knowledgeable, to have access to the resources and social services needed for a decent standard of living and to be able to participate in the life of the community. Without these, many choices are simply not available, and many opportunities in life remain inaccessible”.

So it’s not just about money, drugs and food then (and no, I didn’t think that it was). There’s a lot about the value of people there, and the right of all of us to more than just basic subsistence survival. Which means thinking less in terms of individual vulnerability and starving kids and more about resilience and healthy communities that have the resources to make good decisions and survive shocks with help rather than imperial-style handouts.

The UN has two (at least) main measurements for development: the Human Development Index (HDI : life expectancy, literacy, education, standard of living and GDP) and Human Poverty Index (HPI: gaps in ‘long and healthy life’, ‘knowledge’ and ‘decent standard of living’). These all got rolled up into the Millennium Development Goals (MDGs), a commitment by the UN to reach 8 major development goals by 2015.

So a development reversal would theoretically be something that reverses progress in one or more of these areas. A quick Google for “development reversal” shows the main source of information on these to be the annual UNDP Human Development Reports. And from a quick skim, the 2010 report has a lot of useful background in it that I really need to spend some time reading in depth.

Open source GIS tools – QGIS

I was at the first Data Without Borders Data Dive this weekend, and needed a tool to view and play with some GIS layers. I remembered that this was possible in Open Street Map, and dived into the OSM wiki to find out how. The answer (amongst several other options) was an open-source tool called Quantum GIS (QGIS). Which has been impressing me greatly since I first started playing with it.

Notes to self on this:

    • Loading an OSM map into QGIS needs the OSM plugin (QGIS top menu bar > plugins > manage plugins > Open Street Map plugin) and an OSM map file (there’s a QGIS tutorial on this here – I downloaded the Uganda .osm file from cloudmade).
    • Labels on a map is as easy as double-clicking the layer (QGIS top menu bar> ) and clicking “show labels” in the top left corner of the window.

Loading data from an excel spreadsheet is easy, but you need to enable the plugin for this first (QGIS top menu bar > plugins > manage plugins > Add Delimited Text Layer), then click on the little blue box with commas in it icon that appears in the top icon bar. As far as I know, only csv files can be imported this way, but that’s an easy “save as” from most big excel files.

Loading in a WMS stream (e.g. this http://cernunosat05.cern.ch/arcgis/services/THA/FL-2011-000135-THA/MapServer/WMSServer) is also easy, but you have to remember to click “connect”, then click on the stream names in the window below it, and “” below that.

Development Crises

Back to blogging. I write mainly to keep sane, but also to think through subjects that are complex or I worry that I need to but don’t completely understand yet.

Today it’s development crises. My day job deals with helping to detect large-scale human crises that aren’t natural disasters, but apart from the obvious categories (famine, conflict, poverty limiting access to basic resources like food, medicine, education), it’s not very easy to pin down a) what a non-disaster crisis is, and what is typically done to detect and act to mitigate it. Once I’ve thought about what large-scale human crises are, I also need to think about how the other work that I believe is relevant to them is starting to form a connected space, and what help that space needs from us to keep developing well.

So first. Nomenclature. A quick Google search on “development crisis famine” shows phrases like “social crisis”, “humanitarian crisis”, “famine crisis” and “food crisis”. A wander into Google trends with “development crisis” shows “human development crisis”. Wandering around some on the pages (e.g. a Guardian piece of the impact of global crises on development) shows global drivers like the food, fuel and financial crises, climate change and confidence. Hang on? Confidence? I know that’s a big driver in financial markets, but it’s interesting to think about its role in development.

And examples. There’s a crisis across the Horn of Africa at the moment. The rains failed, crops have failed, livestock is dead and people are starving. But it’s more complex than that, as well explained in this allAfrica.com piece. The guardian piece has a good summary of the IDS Re-imagining Development report  which looked at the complex effects of the 2008 food, fuel and finance crisis interactions. And on a smaller scale, there are the knock-on effects of the natural disasters (that we don’t deal with) on development, for instance the heavy effect that the Eisfjallsjokull eruption had on industries like Kenyan flower-growing  (which relies on air traffic between Africa and Europe).  And then we have this quote in sustainable development: “[A]n environmental crisis, a development crisis, an energy crisis. They are all one.… Ecology and economy are becoming ever more interwoven—locally, regionally, nationally, and globally—into a seamless net of causes and effects.”

So. We have drivers, which can interact with each other and trigger development crises either directly or through causal chains that we may or may not know about yet. And we have crises, and the effects that those crises have on people, environments and social systems (e.g. education and conflict). This sounds very much like a system to me – and one amenable to some serious analysis, sensitivity modeling and risk management. Which must be going on somewhere in the world.