This post will walk through how to stream geotagged tweets onto a Google Map using Node.js. The end result is a real-time heat map depicting where in the world people are tweeting from.

Twitter Map

Step 1 : Coding the Application

The application can be downloaded in full from Github. It has two main parts: the server-side code running on Node.js, and the client-side code running in the browser. The server requests the Tweets from Twitter, parses them and then streams them out via WebSockets to anyone with the web page open. Before we start:

  1. Ensure Node.js is installed.
  2. Download and unzip the project from GitHub.
  3. Open a command window and cd into the directory.

Server-side code

Two files power the server side: package.json and server.js.

Package.json holds a variety of metadata related to the project and lists dependencies.

Server.js is where all of the logic lies. This code first sets up a web server using express, which serves the static web pages, loads socket.io the web socket module, and loads the Twitter API module.

//Setup web server and socket
var twitter = require('twitter'),
    express = require('express'),
    app = express(),
    http = require('http'),
    server = http.createServer(app),
    io = require('socket.io').listen(server);

//Setup twitter stream api
var twit = new twitter({
  consumer_key: '',
  consumer_secret: '',
  access_token_key: '',
  access_token_secret: ''
}),
stream = null;

//Use the default port (for beanstalk) or default to 8081 locally
server.listen(process.env.PORT || 8081);

//Setup rotuing for app
app.use(express.static(__dirname + '/public'));

//Create web sockets connection.
io.sockets.on('connection', function (socket) {

  //Code to run when socket.io is setup.

});

To use the API, you must create an application on Twitter. It’s free and it doesn’t take long. Once you have created your application, load it from the dashboard and then click on API keys tab at the top. The values in the web interface correspond with the following in the JSON:

After that, the interesting part starts:

//Create web sockets connection.
io.sockets.on('connection', function (socket) {

  socket.on("start tweets", function() {

    if(stream === null) {
      //Connect to twitter stream passing in filter for entire world.
      twit.stream('statuses/filter', {'locations':'-180,-90,180,90'}, function(s) {
          stream = s;
          stream.on('data', function(data) {
              // Does the JSON result have coordinates
              if (data.coordinates){
                if (data.coordinates !== null){
                  //If so then build up some nice json and send out to web sockets
                  var outputPoint = {"lat": data.coordinates.coordinates[0],"lng": data.coordinates.coordinates[1]};

                  socket.broadcast.emit("twitter-stream", outputPoint);

                  //Send out to web sockets channel.
                  socket.emit('twitter-stream', outputPoint);
                }
              }
          });
      });
    }
  });

    // Emits signal to the client telling them that the
    // they are connected and can start receiving Tweets
    socket.emit("connected");
});

Let’s walk through the steps.

  1. The client mapping application connects to the web socket server and triggers the connection listener.
  2. Another listener, start tweets, is set up and a connected message is sent to the client telling them they are connected and everything is ready.
  3. When the client receives this message, it sends a message to the start tweets listener and the Tweets get streamed. We only want to stream Tweets with location, so they are parsed, and if they have coordinates we create a simple piece of JSON containing the location. Note we also have to check if there is a stream open, as we don’t want to open one stream per connection (the Twitter API has a cap on the number of streams you can open. From testing, I think it is around 6).

One thing to understand is that every time a client connects, they will create a new connection process. All these connection processes will, however, share the objects outside of the connection listener.

Client-side code

The webpages are all served from the public folder. The index.html file loads the relevant js libraries and then there is a tag for the Google Map. The twitterStream.js is where the logic lies.

  var map = new google.maps.Map(document.getElementById("map_canvas"), myOptions);

  //Setup heat map and link to Twitter array we will append data to
  var heatmap;
  var liveTweets = new google.maps.MVCArray();
  heatmap = new google.maps.visualization.HeatmapLayer({
    data: liveTweets,
    radius: 25
  });
  heatmap.setMap(map);

  if(io !== undefined) {
    // Storage for WebSocket connections
    var socket = io.connect('http://localhost:8081/');

    // This listens on the "twitter-steam" channel and data is 
    // received everytime a new tweet is receieved.
    socket.on('twitter-stream', function (data) {

      //Add tweet to the heat map array.
      var tweetLocation = new google.maps.LatLng(data.lng,data.lat);
      liveTweets.push(tweetLocation);

      //Flash a dot onto the map quickly
      var image = "css/small-dot-icon.png";
      var marker = new google.maps.Marker({
        position: tweetLocation,
        map: map,
        icon: image
      });
      setTimeout(function(){
        marker.setMap(null);
      },600);

    });

    // Listens for a success response from the server to 
    // say the connection was successful.
    socket.on("connected", function(r) {

      //Now that we are connected to the server let's tell 
      //the server we are ready to start receiving tweets.
      socket.emit("start tweets");
    });
  }

This sets up the Google Map then opens up a websocket connection with the server. Once the server confirms it has received the connection and is ready to start sending Tweets, the connected listener is called and the client sends a message back to the server to say it is ready via socket.emit(“start tweets”);. The server responds with a stream of Tweets captured in the twitter-stream listener, where they are added to an array bound to a Google Maps heat layer.

Step 2: Running the Application Locally

First we need to install the dependencies (defined in package.json). From the terminal, cd to the project directory you downloaded so you are in the same folder as server.js. Then do:

npm install

To run the server, do:

node server

You should then be able to open a browser and access http://localhost:8081.

To host the application this we used AWS Elastic Beanstalk. The GitHub read me walks through how to deploy the application.

Improving the Application

How do we take this further? It would be great to dig deeper into the incoming data and rather than just visualizing the data explore trends and gain intelligence. I also wonder what happens if we want to mashup different data sources and read data from other web services or lower-level protocols, such as TCP/IP or WebSockets.

Things get very complicated very quickly when writing code. Here at Safe Software, we don’t think you should have to worry about server-side code to solve your problems—that’s our worry. Here is exactly the same example as above without writing one line of server-side code. All of the workflows and logic are built in a graphical user interface and can then be turned into services and hosted on our cloud platform.

Want to learn more? Watch our free webinar on How to Intelligently Process and Deliver Real-Time Data with FME Server.

Try it Now

We’re here to help you create harmony between your data and applications. We have lots to help ensure your trial is a success including weekly webinars, free training, and full support even during trials.

Start a Free Trial

 

About FME FME Cloud FME Server Twitter

Stewart Harper

Stewart is the Technical Director of Cloud Applications and Infrastructure at Safe. When he isn’t building location-based tools for the web, he’s probably skiing or mountain biking.

Comments

16 Responses to “Using the Twitter Stream API to Visualize Tweets on Google Maps”

  1. Sean Liu says:

    Some questions:
    1) How does your code support running behind a proxy?
    2) How to store the tweets collected? and how to filter them by keywords?

    Thanks,
    Sean

  2. Lucas K says:

    Hi,
    thanks for this great tutorial!

    I am trying to add a keyword filter funtion, since I would like to only show tweets that are geo-located and contain specific keywords. Is there a way to do this?

    As I am new to nodejs, I would be very grateful for any hints. Many thanks!

  3. Stewart Harper says:

    Hi Lucas,

    Check out the Twitter API here that the NodeJS lib is using:

    https://dev.twitter.com/streaming/reference/post/statuses/filter

    We are already making sure that the Tweets are geo-located as we are passing the location bounding box in. If they don’y have an X,Y nothing will be returned. If you want to add a filter too then you need to add the track keyword in:

    twit.stream(‘statuses/filter’, {‘locations’:’-180,-90,180,90′, ‘track’:’Search Term’}, function(s) {

    Or use FME Workbench our product, which just lets you enter the keywords and check a flag for location in a GUI 🙂

    Thanks,

    Stewart

  4. NickSamurai says:

    Hi, amazing tutorial. Thank you so much. May i just ask how to deploy it from localhost. And am also new to node.js. i started the server, but when i go to the page nothing is displayed.

  5. Layla W says:

    Hi,
    Is there anyway to collect about 100MB twits using Twitter API and records the tweet ID, time, key words, and other relevant elements into a DB?

  6. Stewart Harper says:

    I would use AWS DynamoDB or some other NoSQL database to just dump the data in. If you are using Node then you can just write directly to the DB using the AWS SDK: https://github.com/aws/aws-sdk-js/.

    If you are using our product FME, then you can just add a DynamoDB writer to the workspace.

    • Layla W says:

      I just followed your tutorial to build the API. Can you say a little bit more detail on how to use Node to write directly to the DB? THX!

  7. syafiq says:

    A wildcard ‘*’ cannot be used in the ‘Access-Control-Allow-Origin’ header when the credentials flag is true. Origin ‘http://localhost:8000’ is therefore not allowed access.

  8. Yuki says:

    Hello, this article is really useful. But problems happened.
    What I did to the script was only change OAuth setting. And when I went to the browser, I can see the google map but without heat map layer and the console got this message “Failed to load resource: Could not connect to the server.”
    Could you help me with this one?
    Thank you.

  9. PJ says:

    I am trying to run it locally.
    I get the map ui to show up on my localhost:8081 but I can’t see a heat map layer thereafter. What could be the reason?

    Thanks,
    PJ

  10. alelilolu says:

    There is the map but does not show heat map. There is an error throw er something. How to fix that?

  11. Basel says:

    Hi, I think that the heat map will not work anymore now since Twitter deprecated the location object associated with every tweet.

  12. Bruce says:

    Yes the demo is down

  13. Nmir says:

    Hi,

    Now that Twitter has deprecated the location object, is there a way to create a heat map for tweets based on location.

    Thanks..

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts