Tag Archives: Hong Kong

Building an interactive journey time map from data.gov.hk Traffic Speed Map data with Google Maps API

rtm-40-all-region

Some practical applications for road traffic data processing are the subject covered in past installments here, including a traffic jam classifier using GPU deep learning on surveillance camera images and a mash-up of CCTV still image feed with Google Maps. Both are centered around the data source provided by data.gov.hk, which is an initiative to support big data development by disseminating government data of various type to the public through the Internet.

This installment will walk through another mash-up of Google Maps API for real-time interactive journey time map using Google Maps API. Simply put, it is an overlay of real-time road saturation level information on an interactive digital map.  A demo is available here. As a public demo, it is limited to routes from one of three local regions (Hong Kong Island), largely in consideration of the free daily Google API account quota. For personal use, Google’s free daily quota should be more than enough for most needs.rtm-21-colorcode

Work as well in the 3D Earth view.
rtm-12-perspective-arrows-en

Zoom in for a clean view of travel time in color-coded road sections with direction arrows of travel.
rtm-11-morescreens-zoom

Traffic Speed Map data from data.gov.hk

The Transport Department here publishes road travel time data on data.gov.hk for selected road sections. It is not clear from that site the method of collections and collation of this average speed published. However it is envisaged that apart from traditional roadside vehicle sensors and automatic CCTV vehicle speed estimations, public transport operators like buses might have contributed with real time data from their fleet to improve the overall accuracy.

The page at data.gov.hk provided all the necessary information to consume this data feed, including URL to the XML Traffic Speed Map file (which is being updated every 2 minutes), the schema of the XML file, and some supporting documents with the coordinates of the route origin and destination.

As the XML schema suggested, the XML file is structured pretty straight forward. It includes all routes in one single file, under the element jtis_speedmap (JTIS is probably an acronym for Journey Time Indication System, which is a decade old infrastructure aiming at informing motorists, via giant dot matrix displays mounted on highway gantries, estimated travel time of certain routes). In this XML file, each route is represented by one jtis_speedmap node, and underneath this node carries data associated with that route, including the route record identifier LINK_ID.

<jtis_speedmap>
	<LINK_ID>3006-30069</LINK_ID>
	<REGION>K</REGION>
	<ROAD_TYPE>URBAN ROAD</ROAD_TYPE>
	<ROAD_SATURATION_LEVEL>TRAFFIC AVERAGE</ROAD_SATURATION_LEVEL>
	<TRAFFIC_SPEED>27</TRAFFIC_SPEED>
	<CAPTURE_DATE>2017-06-16T18:30:35</CAPTURE_DATE>
</jtis_speedmap>

Most of these data elements are self-explanatory. For more details, there is a separate PDF file describing the data elements in details. In addition, the table available in an Excel file included the origin and destination coordinates of each route.
rtm-06-exceldata

Customary to the local authority’s geodetic practices here, the coordinates in this file are expressed in HK80 grid, therefore translation to Google Maps’ WGS84 system is required. Further details of the translation are explained in this previous post.

Google Maps Directions

Because the data from data.gov.hk only provided the start and end point of each route (indirectly from the accompanying documentations in PDF and Excel), it is not sufficient for color-coded plotting of road sections on digital map – a feature common to most standard GIS visualization application for road network monitoring. In fact, this plotting feature would not be possible without working with complex GIS data structure.

To address this data insufficiency issue, we are going to utilize a publicly available technology, the Google Maps Directions API. With this Google service, routing between two points on road network is as simple as providing the coordinates of the two endpoints. There are many options available from this service, but we are only interested in the “DRIVING” travel mode here for road network.

This route from Google Maps Directions, however, is not guaranteed to represent the exact route the Traffic Speed Map data meant to measure speed for, but should serve well as a reasonably accurate approximation, and can be further fine-tuned manually. For example, route 910-931 will result in a much longer route using Google’s route service than it is more likely to be, as shown in below the “as-is” route from the XML data (left) and the fine-tuned result by making judgement call after assessing the road network (right):

It should be noted that even after fine tuning, there are still some “interesting” routes. But without further information like waypoints from the data provider (the local Transport Department) to resolve these ambiguities, we would settle with this for now.

To make calls to Google Maps API, a Google API Service account is required. In addition, a secret key has to be generated via the Google API Console. By embedding this secret key in any Javascript code, the Google Maps API will be ready for action. There is a good online collection of samples for reference.

Integrating Google Maps Directions with Traffic Speed Map data at data.gov.hk

To stick with the principle of keeping things simple, as in a previous article on Google Maps integration, the target is to keep everything inside one single Javascript HTML file.

The first step is to embed the static data from data.gov.hk into Javascript code. Those data that are not likely to change often are considered static, and in this case, the coordinates of the routes available on the PDF and Excel file previously mentioned. Using simple awk scripts, the following code fragment is generated for such data. The way the Traffic Speed Map file uses hyphens in the link ID necessitated some character substitution legwork during code generation, as this is used as variable names suffix and thus have to conform with Javascript variable naming conventions which prohibits hyphens.

// start dynamic data code generation

// -- route ID 722-50059
origin722_50059= new google.maps.LatLng(22.2859946618326,114.15524436017);
destination722_50059= new google.maps.LatLng(22.28686575617,114.153536622643);

...
// end dynamic data code generation

 

Next thing is to build the method to read the XML file available on data.gov.hk site. For security restrictions to safeguard cross domain issues on modern browsers, the XML file from the remote data.gov.hk server will be mirrored to my web server using cron jobs and monkeyed in order to prevent violation of browser restrictions.

wget -q http://resource.data.one.gov.hk/td/speedmap.xml -O speedmap.xml
gawk "NR==2{print \"<jtis_speedlist>\"}NR!=2{print}" speedmap.xml > speedmap_clean.xml

 

Adding the entry below to the crontab to schedule running the XML file mirroring every 2 minutes for best synchronization to the data update frequency of the Traffic Speed Map at data.gov.hk.


*/2 * * * * /home/ubuntu/syncrtm.sh

 

Now it is time for the XML request method.

	function readXML() {
		var xhttp = new XMLHttpRequest();
		xhttp.onreadystatechange = function() {
			if (this.readyState == 4 && this.status == 200) {
				liveDataXml = xhttp.responseXML;
			}
		};
		// use speedmap_clean for javascript xml parser CORS issues.
		xhttp.open("GET", "speedmap_clean.xml", true);
		xhttp.send();
	}

 

For supporting the extraction of specific data elements from the XML data source, some helper functions are prepared like extractSpeed() and extractSaturation(). These methods used XPath expression, as exemplified below, to retrieve values of interests in the XML.

var path = "/jtis_speedlist/jtis_speedmap[LINK_ID[text()='" + id + "']]/TRAFFIC_SPEED";

 

The TRAFFIC_SPEED value is extracted for displaying along with the LINK_ID as tooltip on each route markers on mouse over. The ROAD_SATURATION_LEVEL value is color coded to red, amber, and green respectively to render the route on the base map. At this point both the static data (i.e. the coordinates of routes) and the dynamic data (i.e. the XML file with speed and saturation) are ready. Plotting the route is simply by calling the Google Maps API “route” method from the class DirectionsService. The callback function holds the response object and will call another method in my implementation to render the routes and markers on the map. This is where our parsed data from the Traffic Speed Map XML file come into play – setting up the color code according to saturation and label the marker’s tooltip with speed information.

directionsService.route({directionsService.route({
  origin: originLatLng,
  destination: destinationLatLng,
  travelMode: 'DRIVING'
}, function(response, status) {
    if (status === 'OK') { // Update the route on screen
        updateRouteDisplay(...);
        console.debug('Route service called and updated display');
    }
....

 

As there will be quite a lot of routes to plot on the same map, and there are overlapping sections according to the Traffic Speed Map data, it certainly make sense to present information with higher value (congested road sections) without being covered by the less valued. A simple trick for this is to set the z-Index of the route to the inverse of its speed. Adding directional arrows along the route also helps to avoid route information being overlooked for overlapping routes as a result of rendering.

var zIndexInvSpeed = 1000 - parseInt(speed); 

directionsDisplay = new google.maps.DirectionsRenderer({
polylineOptions: {
 strokeColor: color,
 strokeWeight: 8,
 zIndex: zIndexInvSpeed,
 icons:[{repeat:'150px',icon:{path:google.maps.SymbolPath.FORWARD_OPEN_ARROW}}]
}
});

 

Event listeners are attached to each of the markers for a simple filtering function. On clicking of a marker, the map will hide every other routes but the one clicked. This will allow some form of interaction for viewers to focus on specific route of interest from a road network full of color-coded road sections.

Another feature is to allow viewers to cycle through overlapping endpoints during filtering. Imagine a position where two routes joined up, that is, where the end of route A is also start of route B. On the map the markers as rendered by Google Maps will certainly overlap. Allowing user to cycle through by simply clicking away greatly improve viewers’ ability to drill down by filtering without being hindered by the overlapping problem. Shown below is an example of clicking on the end of route 930-931 which coincide with the start of route 931-4650. The overall filtering effect will be filtering out 930-931 first, and then filtering out 931-4650, and finally back to no filtering (show all).
rtm-27-cycling-overlapping

The following screen shows the tooltip of a marker, with information of the route ID and the speed from the Traffic Speed Map XML data.
rtm-15-tooltip

Color-coded road sections.
rtm-18-colorcode
rtm-17-colorcode
rtm-26-colorcode

The core of the Javascript file is a loop that will update the map at specific interval according to the Traffic Speed Map XML data. Due to restrictions on request rating imposed by Google Maps API (which is a fair restriction), there are delays deliberately set in my implementation between each DirectionsService route call, even though this has to be done only once on startup for each route.

Finally the secret API key to calling Google Maps API must be included as the parameter to its library, along with the callback method.

<!-- Your GoogleMap API key goes here -->
<script async defer src="https://maps.googleapis.com/maps/api/js?key=YOUR_SECRET_KEY&callback=initMap">
</script>

 

Transportation and traffic analysis is one interest I am particularly fond of.  Studies on travel time has been a serious topic in the academics as well as the transportation industry, and is a very complex cross-disciplinary problem. For the general public, it is still fun to integrate your own free solution and in the way you want it to be.