Use Zato to integrate Django with exchange rate web services in 10 lines of code
(This is a re-post from Zato Blog as Planet Python doesn’t syndicate that one yet)
Summary: The post introduces Zato, an open-source integration platform in Python, and shows you how to integrate Django, or indeed any piece of Python software, with Zato and external web services using nothing but plain Python objects.
Applications in any programming language can be integrated using Zato but being written in Python itself, Zato offer a convenience client for software in Python and that will be used throughout the text.
Zato is a lightweight, yet complete, ESB (Enterprise Service Bus). And the project’s goal is to become a powerful, yet lightweight, one.
Start here for a gentle introduction to what ESB and SOA (Service-Oriented Architecture) are about, but in short, they let you integrate multiple applications each potentially using different formats, protocols and programming languages with the aim of supporting interesting processes you need to automate. And with Zato this is all in pure Python with as little headaches as possible.
How things should stand
As a Python programmer, about the only thing I feel I should need in order to invoke web services exposed by any sort of systems is a simple API based on dicts or other dict-like objects, like Bunch.
It should be always possible to write code like what is below and expect it will just work regardless of the complexity of underlying protocols and data transports.
request = {'from':'EUR', 'to':'HRK'} response = get_exchange_rate(request) print(response.rate) # 1 EUR = 7.4680 HRK as of Jun 13, 2013, 5:00PM GMT |
Given that it’s a blog of the Zato project it won’t come as a surprise that I am about to tell you that Zato allows you to achieve just that, to think in terms of services and dictionaries without having to worry about how everything is actually implemented underneath.
You delegate the job of an actual integration to Zato which becomes the component responsible for dealing with protocols and data formats, fetching information, straightening it and returning to you a unified view. This lets you focus on your job only and nicely follows the UNIX philosophy of separating software into clearly defined blocks interoperating in order to achieve an interesting result. Not to mention that this what the integrations industry has been using to tackle such scenarios for decades now.
This way you can focus on your own app, not on data integration. Someone else takes care of it.
The overall scheme
The diagram depicts what we will achieve:
Users enters a currency code to find EUR exchange rates to in an HTML form
A Django application invokes a Zato client providing a Python dictionary with currencies selected on input.
Behind the scenes, the dictionary is converted into an HTTP JSON call but this is completely transparent to you as a Django programmer.
Zato receives the call already converted to a Bunch instance and invokes 3 web services provided by:
- Yahoo! Finance
- Google CalculatorPseudo-JSON API
- European Central Bank XML API
Output from 3 different sources is converted to a clean Pythonic response sent back to Django
Django app receives a list of dictionaries on output ready to use in a template which is shown to the user
Implementation
Django side
First, clone this repository (we’ll call the directory you’ll clone it to DJANGO_APP_DIR) and run DJANGO_APP_DIR/install.sh – this will use install or upgrade distribute and virtualenv and use pip/buildout to download a couple of dependencies and install everything under virtualenv.
DJANGO_APP_DIR$ git clone git://github.com/zatosource/zato-django-integration.git . DJANGO_APP_DIR$ ./install.sh [snip] DJANGO_APP_DIR$ ./bin/py sampleapp/src/run.py
You can now go to http://127.0.0.1:8188 and witness an ‘[Errno 111] Connection refused’ error. This is OK. Zato is not running yet.
What you can already have a look though is the Django code. Basically, a middleware class is used to inject a Zato client and the client is used to invoke a service which will be defined in the next steps.
Let’s see, this is how the middleware looks like..
from zato.client import AnyServiceInvoker class ZatoMiddleware(object): def process_request(self, req): req.zato_client = AnyServiceInvoker('http://localhost:17010', '/django/sample', ('django-app', 'django-password')) |
from django.template.response import TemplateResponse def home(req): # A dictionary of input data read from HTTP GET. If no input was given # we translate from EUR to HRK. to = req.GET.get('to', 'HRK') request = {'from':'EUR', 'to':to} # Pass the dictionary into the client's invoke method along with the name # of a service you want to invoke response = req.zato_client.invoke('exchangerates.get-exchange-rate-list', request) # response.data has a bunch of attributes that can be fed to the template as is return TemplateResponse(req, 'rates.html', {'data':response.data['rates'], 'to':to}) |
If it were a project where you’d be doing Django programming only then you could congratulate yourself. The code shown above is everything you need to write to invoke a Zato service and fetch the exchange rates.
This is 10 lines of Python code, counting imports or class definitions in. Without the boilerplate, it will be 2 or 3 lines of code needed to invoke web services.
OK, there’s also a trivial piece of HTML, the gist of which is here but that’s it. There is nothing else on Django side, job well done!
Zato side
First thing is, read at least the first part of the tutorial. This will install Zato and create a quickstart cluster. Done? OK, let’s continue. Save the code below as exchangerates.py..
# -*- coding: utf-8 -*- from __future__ import absolute_import, division, print_function, unicode_literals # stdlib from datetime import datetime from traceback import format_exc # anyjson from anyjson import loads # lxml from lxml import etree # Zato from zato.server.service import Service class GetExchangeRateList(Service): class SimpleIO: response_elem = 'rates' input_required = ('from', 'to') output_required = ('provider', 'rate', 'ts') output_repeated = True def get_yahoo(self, from_, to): # Response template out = {'provider':'Yahoo! Finance', 'rate':None, 'ts':None} # Grab a connection by its name conn = self.outgoing.plain_http.get('Yahoo! Finance').conn # Y! Finance needs a query string in that format # ?s=HRKEUR=X&f=snl1d1t1ab url_params = {'s':'{}{}=X'.format(from_, to), 'f':'snl1d1t1ab'} # Invoking the .get method issues a GET request response = conn.get(self.cid, url_params) # Y! gives us a CSV response response = response.text.split(',') # The string we receive is something like # u'"EURHRK=X","EUR to HRK",7.4608,"6/14/2013","5:55pm",7.4629,7.4588\r\n' # and we need the 3rd item. out['rate'] = response[2] out['ts'] = datetime.utcnow().isoformat() return out def get_google(self, from_, to): out = {'provider':'Google', 'rate':None, 'ts':None} # Grab a connection by its name conn = self.outgoing.plain_http.get('Google Calculator').conn # Google needs a query string in that format # ?q=1EUR=HRK url_params = {'q': '1{}={}'.format(from_, to)} # Invoking the .get method issues a GET request response = conn.get(self.cid, url_params) # Convert the pseudo-JSON from # {lhs: "1 Euro",rhs: "7.46464923 Croatian kune",error: "",icc: true} -> # {"lhs": "1 Euro","rhs": "7.46464923 Croatian kune","error": "","icc": true} # so it can be parsed as JSON. json = response.text replace = ('lhs', 'rhs', 'error', 'icc') for name in replace: json = json.replace(name, '"{}"'.format(name)) rate = loads(json)['rhs'].split()[0] out['rate'] = rate out['ts'] = datetime.utcnow().isoformat() return out def get_ecb(self, from_, to): out = {'provider':'European Central Bank', 'rate':None, 'ts':None} # Grab a connection by its name conn = self.outgoing.plain_http.get('European Central Bank').conn response = conn.get(self.cid) xml = etree.fromstring(response.text.encode('utf-8')) ns = {'xref': 'http://www.ecb.int/vocabulary/2002-08-01/eurofxref'} rate = xml.xpath( "//xref:Cube[@currency='{}']/@rate".format(to), namespaces=ns)[0] out['rate'] = rate out['ts'] = datetime.utcnow().isoformat() return out def handle(self): from_ = self.request.input.get('from') to = self.request.input.to for func in(self.get_yahoo, self.get_google, self.get_ecb): try: rate = func(from_, to) except Exception, e: self.logger.warn('Caught an exception {}'.format(format_exc(e))) else: self.response.payload.append(rate) |
and hot-deploy it onto a running server
$ cp exchangerates.py ~/tmp/qs-1/server1/pickup-dir/ |
Both servers will now confirm the deployment, each in its own log (~/tmp/qs-1/server1[2]/logs/server.log):
INFO - Uploaded package id:[1], payload_name:[exchangerates.py]
The service is there but it can’t be used yet.
The way Zato is designed, unless you insist on it your services will never need to directly deal with any addresses, they only need to fetch a connection by its name (‘Yahoo Finance’, ‘Google Calculator’ and ‘European Central Bank’) and its Zato’s job to manage it. You only need to think about overall processes and I/O, not about where an external service to invoke is located. If the location ever changes, you’ll update it using GUI, CLI or API and servers will pick up changes automatically, without any restarts.
Another point to make is that with Zato your code never exposes your own services over any specific transport (HTTP, AMQP and so on). This is also done via GUI, CLI or API.
In fact, if you’re using SimpleIO (SIO), the very same service can be exposed over HTTP/AMQP/JMS WebSphere MQ/ZeroMQ with JSON, XML or SOAP (and CSV is coming soon) without any code changes at all. That depends on what the service does, if it’s a synchronous or asynchronous one but that’s the principle.
Also note that most of the abstractions Zato uses are usually convenience wrappers around best Python libraries out there.
For instance, you can use Python dicts but you can also always use the underlying requests library directly for HTTP calls – you’re never forced to use what Zato believes will be enough for you, there’s nothing preventing you from customizing things to your liking with tools Zato doesn’t offer out of the box.
Likewise, say Zato doesn’t have something by default, like SMTP connections. Given that you’re using Python you can still send out emails in 5 lines of code. (And by the way, SMTP will be added to Zato soon so this will become 1 line of code).
Zato GUI
Let’s fill out a couple of forms in Zato’s GUI to make all the resources need by the service available. Note that it all can be done in JSON and stored in a config repository of your liking but let’s use a GUI here.
Log in at http://localhost:8183 and create a couple of server objects
- HTTP Basic Auth definition
- Plain HTTP channel for Django to invoke
- 3 outgoing plain HTTP connections to
- Yahoo! Finance
- Google Calculator
- European Central Bank
You don’t need to restart server after creating any object.
HTTP Basic Auth definition
Create a new definition and update its password to ‘django-password’ after it’s created – by default passwords are set to randomly generated UUID4s (there are no default passwords in Zato at all).
Plain HTTP channel
Create a new channel object and assign a newly created security definition to it. Note that this particular Python client requires the service to be ‘zato.service.invoke’ and this is the service that invokes the one of yours.
Outgoing plain HTTP connections
An outgoing connection encapsulates information that is to do with particularities of a given transport method. This is everything that a service shouldn’t be concerned with in its own code, such as endpoints, queues, URLs, authentication and so on. Zato deals with it itself, you just need to focus on your own functionality.
Yahoo! Finance
Google Calculator
European Central Bank
Running it all
Now that everything has been created you can visit the Django app at http://127.0.0.1:8188/ and play around with various currencies – this will fetch everything from backend web services and display in an HTML table.
What else is there?
Naturally, this isn’t everything. If you’ve already read the intro to ESB/SOA, you know the first question will be, is the service IRA? Is it Interesting, Reusable and Atomic
- Interesting- sure, if you need exchange rates in your projects such information will be certainly interesting on more than one occasion
- Reusable – almost, the list of providers is hard-coded but ultimately, there should be one or more default provider and client applications should be able to specify which ones they’re interested in
- Atomic – yes, as long as it will be given the feature mentioned above (default providers, client apps say which one to use)
It also makes sense to use Zato’s built-in scheduler and Redis to pre-fetch the rates periodically instead of accessing remote resources for each client request.
The good news is, such things are trivial to add with Zato and once you complete it, you’ll have a truly IRA service that can be reused across a wide range of projects without any code changes. And your client apps will be always able to use plain dicts only.
Can Zato do more?
There’s a whole lot more Zato can do – JSON, SOAP, AMQP, JMS WebSphere MQ, ZeroMQ, Redis, SQL, FTP, load-balancing, scheduling, statistics, hooks, GUI, CLI, API – the features are there.
Note that Django was used in the text but the client is completely framework-agnostic, the same code will work with any Python application.
Also, Zato is in Python but it’s not for integrating Python apps only. As long as your application can speak any of the protocols mentioned (this is 99% of apps out there), you’re good to go.
What next?
If you still haven’t done it yet, read the no-nonsense intro to ESB/SOA and visit the tutorial. This will explain all the core concepts so you can get started quickly.
Thanks for your time! 
