Integrating Third Party Data Feeds

This tutorial isn't specific to ChartIQ but rather a general introduction to implementing third party content. Today, many feeds are available over the HTTP protocol (web based feeds such as REST feeds) however few feeds are yet available that support newer development paradigms ("web 2.0").

The most specific issue has to do with Cross Origin (CORS) restrictions in the browser. Simply put, an HTML page on site aaa.com cannot automatically access data from bbb.com. It's critical to understand that this restriction is the default behavior of browsers. If you want to "mash up" data you need to take specific steps to make it happen.

There are three possible ways around this basic restriction:

1) Add "access-control-allow-origin: *" header to server responses from bbb.com

If a response from bbb.com contains this header then the browser will allow code from aaa.com to use the data. This is the best path to take because it is standards based and the direction for which the web is heading. However, this approach requires that you gain cooperation from bbb.com.

In many cases, API vendors, have security concerns since "*" basically means that anyone on the Internet could then access their API. Mostly such concerns are unfounded, but a cautious vendor be more restrictive by blocking access based on a known list of referrers (that is, their server would allow requests only from "aaa.com").

Unfortunately specifying a domain in the header itself (access-control-allow-origin: aaa.com) does not work well because the standard requires an exact url match. http://aaa.com and https://aaa.com and http://www.aaa.com would each require an exact match, which would require server side code at bbb.com. (The lack of support for wildcards is a severe oversight which we hope the W3C will address in future standards.)

2) Put a Server In-between

Assuming you are unable to get the cooperation of your vendor, the next best alternative is to interject your own server. Instead of accessing bbb.com directly your browser code will need to access aaa.com/yourscript. This server side script (php, perl, python, ruby, node, etc) would then query bbb.com for the data and return it to the browser. (This is how all data integration was done until just a few years ago and still how 80% of the web works).

A quick alternative to writing scripting code is to use a generic proxy server that simply forwards the data (a "reverse proxy"). In most cases this is our recommendation. One would for instance proxy aaa.com/myproxylabel to bbb.com. The data transfer will pass through your proxy server but will obviate the need for the access-control-allow-origin header.

A sample apache http.conf (on aaa.com) would look like this:

ProxyPass /myservice/ http://bbb.com/

3) Use JSONP

JSONP is not so much a protocol as it is a technique, but it bypasses the CORS security restrictions in a browser. Many API vendors offer JSONP based solutions. If you're using jquery then it is very easy to make a JSONP request as the code is interchangable with AJAX requests. This is our recommended approach. Your vendor must explicitly support JSONP however. It is not something you can make work without cooperation of the vendor.