Six Things I Learned Building the Harvey.Geocod.io Twitter Rescue Requests Map

Hurricane Harvey was the first time I’ve ever participated in the civilian response to a natural disaster, and I learned a lot in the process.

My husband and I made a map that showed tweets of people wanting to be rescued (previously housed at harvey.geocod.io, but since taken down out of respect for victims’ privacy). By the time Harvey left the Texas coast, it had logged over 1,800 requests for help.

With Irma continuing her course up the Southeast coast, I wanted to share what I learned about developing for disasters and participating in volunteer-led response so others might feel empowered to use any relevant skills they might have to help with this and other future disasters. (Warning — this is long.)

1. It didn’t occur to us immediately that we had useful skills.

Starting on Saturday, August 26, we started seeing tweets like this. It was overwhelming to see that people were unable to get through to 911 and so, out of desperation, they were tweeting their full addresses at local TV stations and Houston city government accounts hoping someone would see it and rescue them.

Watching a disaster unfold in real time in excruciating detail like this was heartbreaking.

Initially, my response was the same as it has been with previous disasters: figure the best I could do was donate money, then still wish I could do more.

I spent most of that weekend glued to Twitter (even though, earlier that week, I’d deleted the app from my phone to try to wean myself off of it). On Sunday afternoon, it occurred to me that it might be possible to make a map the tweets, but I wasn’t 100% sure — and besides, someone had probably already done that, right?

It wasn’t until Sunday night — after taking the only extended Twitter break of the weekend to watch the Game of Thrones finale— that it occurred to us that we not only might be able to do more than donate but that we needed to. I mention this because often in a movie or story when an unlikely participant jumps in, it’s evident to them immediately; in our case, it took some time to bake. I think the time away from watching the situation was a necessary step in the process, because after taking that brief break back to normal life, I felt the situation more intensely and urgently. Scrolling through those threads earlier had hit me like a brick, but now I broke down and cried.

I found myself quickly searching to see if anyone had made maps that were automatically pulling the tweets to the various hashtags. There weren’t any. Meanwhile, I’d started texting with a close friend who was also following the situation on Twitter and shared the map idea I’d had earlier in the day. It wasn’t until this point that it occurred to me that even if we didn’t know it was possible to map the requests or if we’d be able to get it in the hands of rescuers, we had technical skills that could lead to a simple prototype (prior experience working with Twitter data and our own geocoder to to turn them into lat/lon and map them), and we therefore had the responsibility to at least try.

I started to work through what we’d need in my head, and a few minutes later, my husband Mathias and I crawled into bed with our laptops and got to work.

2. Technical feasibility is not a concern when you’re motivated by the problem.

When we first sat down to work on it, we had no idea if it was even possible. Were people tweeting complete-enough addresses? If not, how might we complete them? Where would we find them, since Twitter’s basic API won’t let you do a blanket query? How would we display them? What kind of format would the end user need?

Despite the emotion of the situation, habit kicked in and we were able to get to work. First, we had to figure out what we could get from Twitter; we knew from past experiences that we could only get the firehose from a data provider like Datasift or Gnip; we’d need to filter by hashtag or accounts. So Mathias got to work making a basic Laravel app and connecting the Twitter API (thanks, Freek Van der Herten, for the open source Twitter package!), and I started making a list of all of the hashtags and accounts that people were tweeting at with their addresses.

Compiling that list was overwhelming. People were tweeting at local TV stations, the newspaper, the mayor, hoping anyone might see it and come rescue them. In a story the Boston Herald did about the map, I’m quoted as saying I’d never cried as I built something before, and it’s true. I’m not a crier, but it was impossible to get through those tweets without welling up. We ended up having over 50 hashtags and accounts we scanned for tweets.

Next: filtering out the tweets with addresses. We noticed there was generally a pattern to the addresses: usually at a minimum there were several numbers in a row (perhaps a zip code or house number), and maybe a word that is synonymous with “street” or a city name. We already had a long list of street suffixes from Geocodio, so all we needed to get were the numbers and make an assumption that they were in the general vicinity of Houston. The resulting filter that parsed tweets looking for addresses ended up being about 500 lines of code.

There was a brief celebratory moment when we finally got the parser working. It quickly ended, though, when we saw the content of the tweets coming in.

The broader lesson I took from this is that when everyone working on something is deeply motivated by the problem being solved, uncertain technical feasibility isn’t a blocker. I’ve often run into situations at work where less-than-90%-certain technical feasibility is the fast route to the “archive” pile in Trello, but this showed me that if the desire to solve the problem is strong enough, people will find a way. It’s made me question the received dictum for product managers that we’re wasting developers’ time if they’re not sitting in front of a screen coding, and has put urgency into my own efforts to enable them to interact or observe our users closely on a regular basis.

3. Do the bare minimum, on purpose.

With the hard part done — we had the tweets, and we’d converted them into addresses — we next had to figure out what to do with that information. Having no prior experience in disaster response, we had absolutely zero idea what the end user would find useful, so we built the simplest and most flexible tool possible. So we open sourced all of the code and made a simple map and a JSON feed. Maybe someone would be able to find one of those useful or build off of what we created.

We thought about whether we should add additional features — filtering by time or location, or the ability to mark someone as rescued — but wanted to get it in the hands of people who could have informed opinions about those needs first. So we quickly threw together an email to anyone on our Geocodio customer list with an email address at a relief organization or anything remotely related to Texas, hoping someone would be able to connect us with rescuers to see what they might need.

Throughout that week, we were reached out to by several people from the Red Cross — they’d used our geocode-a-spreadsheet tool during volunteer trainings — and gave them our data so they can use it for density mapping of the hardest-hit areas for relief operations.

4. Building when lives are potentially at stake is an utterly insane level of stress.

I mentioned I cried while pulling together the list of hashtags and accounts people were tweeting at, and it wasn’t the last time I cried during this, either. This is without question the most emotionally trying and exhausting project I’ve ever worked on.

I found it impossible to look through the map myself without having a sinking feeling in my gut as I looked through all of the tweets. There were new ones, all the time. Each one wasn’t just one tweet: it was often families, with young children and pets. Or entire apartment buildings and retirement homes full of people — one tweet I remember said there were 70 people at one address that needed to be rescued. 70 people. It’s difficult to test and improve a product when you can only look at it for a few minutes before having to look away and do something else.

But keeping an eye on the map was critical. The situation on the ground changed constantly, and that required us to make on-the-fly changes. On Tuesday, when the storm spread to Beaumont and Port Arthur, we realized we needed to change the address parser since it had assumed that addresses were in the Houston area. First thought: oh, god, have we missed people by having the filter too tight? Mathias quickly changed the algorithm during his lunch break — and we realized later that there had been a bug in it and it was only looking at Port Arthur. (Oh, god, have we missed people by having the filter too tight?, x2). I’ve never before had a code change potentially have life-or-death consequences before, and I felt guilty for asking him to rush to change it to incorporate more people.

The whole experience taught me that disaster response nor the medical field are fields I should consider entering. The stress of a product needing to function perfectly lest peoples’ lives might be in danger was just utterly insane and I admit I didn’t handle it well.

5. I don’t know if we actually helped anyone.

And that gets me to the next point… might. Peoples’ lives might have been in danger because of the code change we made. But we don’t know for sure. We don’t actually know for sure whether the map we made was used on the ground by rescuers or resulted in the rescue of a single person. We had several rescue groups, like Project Rubicon, ask us if they could use the map, and volunteers with the spontaneously-organized global effort to track victims that recently evolved into CrowdRescueHQ used our data to double-check their records that no one had been missed, but we don’t know for certain whether anyone was rescued because of the map we made.

I didn’t, and still don’t, handle this well. After the Boston Herald story and one for our local news station, we had a lot of people tell us how proud they were that we were doing this. While I recognize people were just trying to be nice and may have genuinely felt that way, both of us felt a bit uncomfortable hearing that. People were praising us when we didn’t know if we’d actually helped anyone. It felt gross, and I ended up getting somewhat defensive after the first couple of people said something, replying something along the lines of, “I just made a map. I didn’t actually get in a boat and rescue anyone.”

Someone told me that they were looking at the map at work and another person looked at the map and said “This is brilliant!” I can’t find it brilliant. I found, and still found it, heartbreaking. People have been asking us if we could build the same map for Irma, and my reply is probably, but I hope not. We were only able to build this map because people were so desperate to be rescued that they broke the normally-strong social convention to not share their addresses and phone numbers on the internet. I wish we hadn’t needed to make the map in the first place, and I hope we won’t need to for Irma.

6. The Internet is still a force for good.

Lest I end on a down note, there was one other strong takeaway from all of this: the Internet is still a force for good. Over the last ten months, I’ve often found myself questioning whether the Internet and technology are net positives in the world. I especially questioned Twitter, which went from a harmless app used to share what you had for lunch to a place that allowed all sorts of hateful fringe ideologies to unite and was generally a melting pot of negativity on any topic.

But even though the Internet brought out the worst in people for the first nine months of this year, it brought out the best in them for the last week of August.

CrowdRescueHQ, which emerged out of other people on Twitter trying to connect rescue requests with boaters, grew to be a Slack team with over 700 people around the world. Throughout the week we were in regular contact with people from California to Holland who were using the tweets we were gathering to coordinate rescues. To be part of a decentralized global volunteer group was chaotic and frustrating and beautiful and amazing, and it never would have happened without the technology we have today.

And for us alone, we were able to scrape together a simple app to try to help in any way we could all the way from Virginia. True, we wished we’d be able to just get a boat and go down there ourselves, but we didn’t have to because of the Internet. (And having no experience with water rescues, that was probably for the best.)

This entire experience restored my faith in the power of the Internet, and more importantly people plus the Internet, to make positive contributions to the world. I ended up sharing the internal conflict in the section above with a friend last week, and she said something resonant: “It doesn’t matter if you actually rescued someone. You showed other people that normal people like them can do things, can try to help. You inspired them, and maybe next time they’ll be able to use skills that they didn’t realize were relevant to help. You gave them the permission to try.”

I really hope that ends up being the case, and that someday when a disaster strikes and you realize you have skills that are kinda-maybe?-sort-of-possibly helpful, you’ll feel like you have the permission to at least try to pull something together.