Maps: Turning numbers into people
Why we love maps, and how we build ours using D3
We love maps at Socratic.
Our team of 6 was born and has lived all over the world — 5 continents, 10 countries, 8 languages — so we’ve all grown up looking at maps and feeling connected to people from far away places.
We knew early that we wanted to have maps all over Socratic. It wasn’t just about our personal love for maps — they also helped us achieve some important product goals:
- Teachers contribute to Socratic to reach students beyond their classroom walls, and students contribute to give back to the people that have helped them (among other motivations). Maps are a natural way to show users their global impact.
- When a user starts contributing to Socratic, the number of students helped in their first few days is, naturally, low. “25 views” isn’t impressive on a dashboard, but 25 dots in 6 countries on 3 continents packs a much heavier emotional punch.
- We believe that a platform like Socratic can certainly teach people the academic lessons they need, but can also indirectly teach some other important lessons: Knowledge builds on itself. We’re all students and teachers. People all over the world have similar struggles and desires. We’re not so different from one another.
World maps are one of the few ways we’ve tried to get these ideas across.
These ideas come together on a Socratic contributor’s profile page. At the very top, we show a full-width world map, with a dot for every place the user’s contributions have been seen.
So, we love maps. Now how do we make them?
We started with some short-term and long-term requirements. What we wanted was:
- Lots of dots! Users accumulate a ton of views from literally every country in the world (surprisingly, including North Korea), and we didn’t want to limit the detail on their maps. We wanted to be able to show tens of thousands of dots with tooltips smoothly on most devices and browsers.
- Custom colors for the land, oceans, location markers, and coastlines, per map.
- Custom borders. Sometimes we want to show country borders, other times we want to show just the land/sea border. Sometimes states and provinces.
- Custom markers tooltips, with arbitrary HTML/CSS/JS.
- Save as PNGs. We don’t just want dynamic maps on the site, we also want static maps in emails, which means saving rendered maps as .png images.
We explored Google Maps, and Leaflet + Mapbox, but neither fulfilled all our requirements, so one morning, inspired, we built our own solution.
First, make it work.
We’d had a hunch that D3.js would be the right tool for this job.
“D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG, and CSS.”
D3 lets you take raw JSON data and do a seemingly endless number of beautiful, interesting, interactive things with it. It’s always fun to see what Mike Bostock and others imagine up with D3, and I knew I’d seen maps made with it too. Sure enough, I soon found a guide for making D3 maps.
Distilled, there are a few fairly simple steps (and here’s a repo with a working example).
(1) Find landmass shape data.
If we’re going to render a map, we need to know the shapes of coastlines and borders. I followed the instructions in the guide above, converting Natural Earth’s 1:110m land polygons (because I wanted to show the whole world) to TopoJSON. The exact steps I took are in this gist. The final TopoJSON file is available here.
(2) Create an SVG element to hold the map and dots.
D3 deals with SVG, a vector graphics format where shapes are defined in XML, and is great at creating and manipulating SVG shapes and paths.
Assuming there’s a <div> on the page with id `map`,
var svg = d3.select(‘#map’).append(‘svg’);creates an SVG element within the div, which you can manipulate using D3.
(3) Load the TopoJSON data so D3 can use it.
You’ll need two external scripts (and a third if you’re using a map projection that isn’t included with D3, as I did).
Add `https://cdnjs.cloudflare.com/ajax/libs` before each of these:
<script src=”/d3/3.5.5/d3.min.js"></script>
<script src=”/topojson/1.6.9/topojson.min.js"></script>
<script src=”/d3-geo-projection/0.2.9/d3.geo.projection.min.js"></script>Then load the TopoJSON data in another <script>:
var svg = d3.select(‘#map’).append(‘svg’);d3.json('static/json/landtopo.json', function(error, geoJson) {
// The shapes of the continents are loaded into `geoJson`.
// Everything else happens here.});
(4) Render the world!
This block of code loads our TopoJSON file and renders the shapes defined in it within the `svg` we created in (2).
// select a projection
var projection = d3.geo.miller();// create a geographic path generator
var path = d3.geo.path().projection(projection);
// use the generator to draw shapes defined in our json data
svg.append(‘path’)
.datum(topojson.feature(geoJson, geoJson.objects.landgeo))
.attr(‘d’, path);
This will render the outline of the shapes in the TopoJSON file, which shows the world’s continents.
(5) Place dots on the map
We’ve got a world map, so let’s put dots on it! Suppose we’ve data on some of the places our team has lived:
mapData = [
// [longitude, latitude, num_years, location_string]
[18.9500, 50.3833, 13, "Piekary Śląskie"],
[-104.9903, 39.7392, 14, “Denver”],
[106.8000, -6.2000, 11, “Jakarta”],
[-74.0059, 40.7127, 4, “New York”],
[-79.4, 43.7, 4, “Toronto”],
[73.02, 26.28, 1, “Jodhpur”],
[-122.1700, 37.4300, 4, “Stanford”],
[8.5500, 47.3667, 2, “Zurich”],
[-99.1333, 19.000, 2, “Mexico City”],
[-47.8828, -15.7939, 2, “Brasilia”],
[32.5333, 15.6333, 5, “Khartoum”],
[-96.6992, 33.0197, 5, "Plano"]
];D3 lets us go through each item in the array, create a circle for it, calculate the x-offset, y-offset, and the radius of the circle, and add the circle to the page.
The x-offset and y-offset are calculated by applying the `projection` function to the longitude or the latitude. The radius is the result of any function we wish, here, twice the square root of the `num_years` at that location.
The syntax for this involves the `selectAll`, `data`, `enter`, `append` sequence common in D3. This and this helped me get the hang of it.
svg.selectAll(“circle”)
.data(mapData)
.enter()
.append(“circle”)
.attr(“cx”, function(d) {
var cx = projection([d[0], d[1]])[0];
return cx;
})
.attr(“cy”, function(d) {
var cy = projection([d[0], d[1]])[1];
return cy;
})
.attr(“r”, function(d) {
return Math.sqrt(d[2]) * 2
});This places all the dots in our data on the map! This was an exciting moment. I was thrilled that code this simple could produce such a powerful result.
(6) Style it
The beauty of SVG is that elements can be styled just like HTML, with CSS. This customizes the colors of the land, sea, coastlines, and circles:
/* map container */
svg {
height: 420px;
width: 950px;
}
/* land outline */
svg path {
fill: #00AEA0;
background: #00AEA0;
stroke: black;
}/* map markers */
svg circle {
fill: white;
stroke: white;
}
Boom, we’ve got a pretty map!
Then, make it fast.
That was it for v1 of our map. We shipped it, and rejoiced. One day of work, and users loved it. You can see the basic v1 code in this repo.
A few months in, our top contributor had accumulated close to a million views, which, after clustering views at the same location, amounted to ~20,000 dots. Displaying them would take enough time on a decent computer to noticeably freeze the page, block scrolling, and generally be a bad experience.
The first thought was to write a function that rendered a few hundred circles at a time, with `setTimeout` at the end, calling the function again with a slight delay — a classic animation pattern.
But we wondered, is `setTimeout` really at the center of all the fancy animations on the web? It seemed like too blunt a tool — with few timing guarantees, and not optimized for what else is happening in the browser.
Turns out, browsers have implemented an almost drop-in replacement for `setTimeout`, called `requestAnimationFrame`. This tells the browser that we’re trying to animate something, and the browser then repaints the appropriate part of the page at the right frame rate, usually 60 times a second. It also optimizes for whether the element is visible, whether the tab has focus, battery status, etc. We found a little piece of code called `renderQueue`, which takes a queue of data and a render function, and renders each piece of data using the browser-specific implementation of `requestAnimationFrame`.
A few lines of code later, we had the current map — animating 100 dots at a time, and working nicely with the browser so the page never freezes. What’s better, it looks way cooler!
What’s next?
We aren’t yet using this solution everywhere. We have a few maps in emails, where we can’t run Javascript and have to use images. We use Mapbox to generate .jpgs for these, which comes with a few limitations we don’t like.
So next we’d like to build a map-image service. We’d want to keep using D3, so it’ll likely be a small node.js app (which is not yet part of our stack). It seems simple to generate a PNG from SVG and store it on S3, and we could do some basic hash-based caching of incoming requests to keep things zippy.
But, we’re a small team with a bunch of other big projects on our plate — our first iOS app which will make it super easy to find the help you need, a diagram creator to make force diagrams in Physics, demand curves in Economics, etc., moderation tools for the community, tools to make sense of all the quality signals students provide, and on and on — so this will likely only happen after we hire the next two engineers, who we’re looking for right now!
If you love maps, education, or the idea of building a community that empowers people all over the world, we’d love to hear from you.
Thanks
http://bost.ocks.org/mike/map/
http://bl.ocks.org/syntagmatic/raw/3341641/
http://www.paulirish.com/2011/requestanimationframe-for-smart-animating/
http://creativejs.com/resources/requestanimationframe/
The Earth View Chrome extension, which loads beautiful and random satellite images into new tabs, with Google Maps links to the locations.
