Coding, How-To

How to Create Geo HeatMaps with Pandas Dataframes and Google Maps JavaScript API V3

Get excited because we’re going to make a heatmap with Python Pandas and Google Maps JavaScript API V3. I’m assuming the audience has plenty of previous knowledge in Python, Pandas, and some HTML/CSS/JavaScript. Let’s begin with the DataFrame.

The DataFrame

First, you’re going to need a dataframe of “addresses” (can be a physical address, or even just a country name, like USA) that you eventually want to plot. (For the sake of simplicity, I’ll try to refer to the “address” as the “geo” for the rest of this document.) Second, since you are planning on using a heatmap, you’re going to want some sort of number that represents the weighted value of that row in comparison to other rows.

Let’s say your DataFrame looked like this:

grouped_country_df = main_df.groupby('country')\
                            .agg({'pink_kitten': lambda x: len(x.unique())})\
                            .sort('pink_kitten', ascending=False)
print grouped_country_df
geo_name count_of_pink_kittens
USA 3430
Spain 577
United Kingdom 352
Israel 292
Austria 196
Argentina 151
India 133
Singapore 66

Now you have a list of geos and some values to use as the weight when later creating the heatmap. But to plot these points, you’re going to need some lat and long coordinates.

Getting Lat Long Coordinates from Google Maps API

If you have a list of geos or “addresses,” you can use Geocoding to convert those geos into lat/long coordinates. From Google: “Geocoding is the process of converting addresses (like “1600 Amphitheatre Parkway, Mountain View, CA”) into geographic coordinates (like latitude 37.423021 and longitude -122.083739), which you can use to place markers on a map, or position the map.”

To use this Google Maps service, you need to have a Google Maps API key. To get a key, you can follow the directions here. When you sign up for an API key, you should select “Server Side Key,” since we will be running a Python script server-side to access the Google Maps API.

Once you have your api_key, you can work on getting geocoded results for all of your geos. You can do this with the following code:

import requests
# set your google maps api key here.
google_maps_api_key = ''

# get the list of countries from our DataFrame.
countries = grouped_country_df.index
for country in countries:
    # make request to google_maps api and store as json. pass in the geo name to the address 
    # query string parameter.
    url ='https://maps.googleapis.com/maps/api/geocode/json?address={}&key={}'\
         .format(country, google_maps_api_key)
    r = requests.get(url).json()

    # Get lat and long from response. "location" contains the geocoded lat/long value.
    # For normal address lookups, this field is typically the most important.
    # https://developers.google.com/maps/documentation/geocoding/#JSON

    lat = r['results'][0]['geometry']['location']['lat']
    lng = r['results'][0]['geometry']['location']['lng']

This only gets you so far, since you still need to do something with those latitude and longitude coordinates. We have a few options here:

  1. If you are building a web application, you can pass those values into an HTML template as variables and they will end up getting plotted via JavaScript.
  2. We can print out the format of the JavaScript, and later past it into our HTML file within script tags.
  3. Other approaches that I’m not going to talk about.

For the sake of time, I’m going to show #2, which lends itself to a one-off analysis. You’d probably want to go with some dynamic templating approach, like #1, if you are going to pull and plot the same data repeatedly.

Add the following code to your for-loop from above, right underneath

lng = r['results'][0]['geometry']['location']['lng']

# set the country weight for later. by getting the value for each index in the dataframe
# as it loops through.
country_weight = int(grouped_country_df.ix[country])
 
# print out the Javascript that we will be copy-pasting into our HTML file
print '{location: new google.maps.LatLng(%s, %s), weight: %s},' % (lat, lng, country_weight)

After running your script, copy the output, which should look like this:

{location: new google.maps.LatLng(37.09024, -95.712891), weight: 3430},
{location: new google.maps.LatLng(40.463667, -3.74922), weight: 577},
{location: new google.maps.LatLng(55.378051, -3.435973), weight: 352},
{location: new google.maps.LatLng(31.046051, 34.851612), weight: 292},
{location: new google.maps.LatLng(47.516231, 14.550072), weight: 196},
{location: new google.maps.LatLng(-38.416097, -63.616672), weight: 151},
{location: new google.maps.LatLng(20.593684, 78.96288), weight: 133},
{location: new google.maps.LatLng(1.352083, 103.819836), weight: 66},

You’re going to use these values in the next step.

Creating an HTML file that contains Javascript for Plotting your Lat Long Points.

You need to create an HTML file that contains some script tags within it. I am simply going to paste my code below with annotations. If you copy the location strings from above, you will be able to paste them directly into this HTML file under the “heatmapData” array (defined below in the code).

<!DOCTYPE html>
  <head>
    <title>Simple Map</title>
    <meta name="viewport" content="initial-scale=1.0, user-scalable=no">
    <meta charset="utf-8">
    <style>
      html, body, #map-canvas {
        height: 100%;
        margin: 0px;
        padding: 0px
      }
    </style>
    <!-- Load Google Maps API. -->
    
    
  
    
    function initialize() {
      var heatmapData = [
        {location: new google.maps.LatLng(37.09024, -95.712891), weight: 3430},
        {location: new google.maps.LatLng(40.463667, -3.74922), weight: 577},
        {location: new google.maps.LatLng(55.378051, -3.435973), weight: 352},
        {location: new google.maps.LatLng(31.046051, 34.851612), weight: 292},
        {location: new google.maps.LatLng(47.516231, 14.550072), weight: 196},
        {location: new google.maps.LatLng(-38.416097, -63.616672), weight: 151},
        {location: new google.maps.LatLng(20.593684, 78.96288), weight: 133},
        {location: new google.maps.LatLng(1.352083, 103.819836), weight: 66},
      ];
       
      // Add some custom styles to your google map. This can be a pain. 
        // http://gmaps-samples-v3.googlecode.com/svn/trunk/styledmaps/wizard/index.html
      var styles = [ 
        {
          "featureType": "administrative",
          "stylers": [
            { "visibility": "off" }
          ]
        },
        {
          "featureType": "road",
          stylers: [ 
            { "visibility": "off"}
          ]
        },
        {
          "featureType": "landscape",
          "elementType": "geometry.fill",
          "stylers": [
            { "color": "#ffffff" },
            { "visibility": "on" }
          ]
        },
      ];
      // create a point on the map for the Atlantic Ocean, 
      // which will later be used for centering the map.
      var atlanticOcean = new google.maps.LatLng(24.7674044, -38.2680446);
      // Create the styled map object.
      var styledMap = new google.maps.StyledMapType(styles, {name:"Styled Map"});
      // create the base map object. put it in the map-canvas id, defined in HTML below.
      map = new google.maps.Map(document.getElementById('map-canvas'), {
        center: atlanticOcean, // set the starting center point as the atlantic ocean
        zoom: 3, // set the starting zoom 
        mapTypeControlOptions: {
          mapTypeIds: [ google.maps.MapTypeId.ROADMAP, 'map_style'] // give the map a type.
        }, 
      });
       
      // Create the heatmap object.
      var heatmap = new google.maps.visualization.HeatmapLayer({
        data: heatmapData, // pass in your heatmap data to plot in this layer.
        opacity: 1, 
        dissipating: false, // on zoom, do you want dissipation?
      });
      heatmap.setMap(map); // apply the heatmap to the base map object.
      map.mapTypes.set('map_style', styledMap); // apply the styles to your base map.
      map.setMapTypeId('map_style'); 
       
      // Add a custom Legend to Your Map
        // https://developers.google.com/maps/tutorials/customizing/adding-a-legend
      var legend = document.getElementById('legend');
      map.controls[google.maps.ControlPosition.RIGHT_BOTTOM]
         .push(document.getElementById('legend'));
       
      // This is hard-coded for the countries I knew existed in the set.
      var country_list = ['USA','Spain','United_Kingdom','Israel',
                          'Austria','Argentina','India','Singapore'];
       
      // for each country in the country list, append it to the Legend div.

      for (i = 0; i < country_list.length; i++) {
          var div = document.createElement('div');
          div.innerHTML = '<p>' + country_list[i] + '</p>'
          legend.appendChild(div);
      } 
    }

     google.maps.event.addDomListener(window, 'load', initialize);

</script>
</head>

<body>
    <'div id="legend" style="background-color:grey;padding:10px;">
    <strong>Countries Mapped</strong>
    </div>

    <'div id="map-canvas"></div>
    </body>
</html>

Open the HTML file in your browser, and you should see something like this.

google maps heatmap

Et Voila!

Standard
Coding, Hackathon

Winning 2nd Prize (Place?) at Music Apps Hack Weekend!

This past weekend, Spotify and a few other brands put on an amazing hackathon at SPiN NY. Yes, there was ping pong, and in fact, there was a ping pong tournament. Other things that were there: lots of Mountain Dew, lots of alcohol, lots of Doritos and lots of McDonald’s. All four of those items helped everyone get through the weekend, and they were there due to the epic sponsorship of the whole event.

The event kicked off on Friday night with a live performance from “Blood Orange,” which turned out to be really awesome. I showed up the next day around 9 a.m. for the breakfast, which was provided by McDonald’s. McDonald’s did a good thing and gave all of the leftovers to a shelter.

As I was sitting down reading about the Spotify API and trying to figure it out, a guy sat down next to me. He started talking with a friend of mine — and my friend told the mysterious gentleman that I could use some help (it was true). Spotify’s API was all JavaScript, and I’m not as familiar with OOP JavaScript as I wish I was. Thus, I was introduced to Toby at about 11:30 a.m., and we started working together at about 12:00 p.m. (late start).

We were setting out to build the Spotify Apps game. The idea initially came from my girlfriend, Tal. She told me about Apples to Apples, the boardgame, and we discussed tweaking it so that it was a music game. I thought it was a pretty good idea. And we worked out a few of the game mechanics on Friday night. As such, the core concepts of the game were known by Saturday. However, before I met Toby, I was considering working on something else that was a bit easier, I must admit.

But I did meet Toby, and we started hustling — setting up a GitHub repository and planning out the project in a Google Doc. At around 2 or 3 p.m., Tal showed up. She lent a hand on the user experience and game flow. She also began to design some of the core elements. We had a functional, working name by about 4 p.m., but none of us were completely sold on it: “Songville with Friends.” It was more of a joke in the beginning, and it was one that we thought would help us earn a laugh or two from the crowd. However, after an evening brainstorm, we ended up changing our name to SongJitsu. At that point, we had a good handle on the branding: Ninjas. And that branding is awesome. 

Tal began to redesign the pages in accordance with the many game states (complicated? check.). Meanwhile, as the designs came in, I was coding them on the frontend. Toby was working on the backend, building it out in Ruby. I had some experience with Ruby in the past, but I didn’t think I was going to be much help to him, so I kind of stuck myself on the frontend coding (there were 570 lines in styles.css). I did a quick count on some of our code, and we have over 1,500 lines (that doesn’t count rewrites, haha… ha… CRY).

We kept kicking ass throughout the night, and were, overall, very impressed with the event. Around 9 p.m. on Saturday they held a ping pong tournament, but before playing, there was a demo from two pros. They were AMAZING. And one of them even stripped down to his spandex underwear.

Around 9:30 p.m. or so, Doritos came by and informed us that we had won the services of a Doritos intern from the hours of 10 p.m. to 9 a.m. on Sunday. Additionally, we had $375 to spend on anything we wanted, except for booze and drugs, and we could send the intern to do anything for us. Wise as we are, we sent the intern to go find us ninja costumes for our presentation. Nothing was open, so she fashioned us some red bandanas, which we ended up losing anyway. Either way, it was a really neat idea that Doritos came up with, and it kept things interesting.

Code. Code. Code. Somewhere between 3 and 5 a.m., McDonald’s ordered pizza for everyone. This was humorous. It was greatly appreciated, slightly confusing, and unsurprisingly, it wasn’t Domino’s. Instead? Fat Sal’s.

Code. Code. Code. 5 a.m. Things were coming together. The app had an interface, Tal was telling me to move pixels, I told her to go to sleep (she did and I changed the pixels anyway).

“Code.” “Code.” “Code.” It’s 7.am now. Tal is sleeping, but about to be woken up. Toby and I aren’t writing our greatest code anymore. Sometimes I just stared at the screen. We ask each other questions and sometimes there is no answer.

7:30 a.m. I wake Tal up.

“Let’s all take a moment to examine the horrific posture in this room.” — Toby at 7:35 a.m. I try to sit up.

9 a.m. now, and I’ve hit the 24-hour mark. We have 1.5 hours left till the defined “stop coding” time. We still have a lot of things to work out, specifically, connecting to the Spotify API and getting the player working in our app.

10:30 a.m. Still coding. They blow a whistle or something, and everyone pretends they didn’t hear it. We still haven’t gotten the player working.

10:45 a.m. The player works! I finalize a JS/jQuery function that runs when a player wins. The function name: win_and_slide();

12:00 p.m. We’re actually done. The app works for the demo. I’m tired. Toby is tired. Tal is tired. The whole room is tired, and people are getting set up to present.

We’re on deck at some random time. Between 12:00 p.m. and 3:00 p.m. I didn’t have a watch on and forgot about my phone. I start thinking of everything in terms of an #id or .class.

We get on stage, present, and things go well. Then we wait. The judges go to vote and prize-giving ensues. Some people win DVDs and Mountain Dew skateboard decks. They announce that the $5,000 prize goes to: “SONGJITSU.” Tal starts screaming and jumping up and down. In my head, I call the function win_and_slide();. Toby is stoked — big group hug, lots of smiles. All of that teamwork paid off.

We collect the oversized check and take photos with it, and we admit that we will probably be eating Doritos for the rest of our life.

After the event, I head to go grab some food with Tal at a diner. I fall asleep at the table and am virtually sleeping while walking. It was around 5 or 6 p.m. by then and I’m pushing being up for 32 straight hours. We get back to Tal’s place and I fall asleep in about 30 seconds.

I woke up today at 8:30 a.m. and got ready for work.

Standard