András Veres-Szentkirályi has started a series on securing web services with Python; the first part deals with UserNameTokens and mentions sec-wall, the security proxy, yay! 
I’ve gotten a little bit tired of long constant names, like imagine there’s constants.py module which has the following:
MESSAGE_CREATE_FOO = '1000' MESSAGE_DELETE_FOO = '1001' # And so on..
Now the trouble is with importing it all. I can do either:
from constants import *
Which may be considered a bad style on one hand and on the other hand it becomes a mess if your IDE, like Wing IDE, autocompletes the names for you, so you type ME and then have to scroll through several dozens of them. Not to mention that it’s hardly ever the case that all of the constants are needed in any single module.
I can also carefully pick the names I need, like so:
from constants import MESSAGE_CREATE_FOO, MESSAGE_DELETE_FOO # And so on..
But that’s yet another place that needs to be kept in sync with the rest of the code.
So I’ve finally settled on using the Bunch class, which basically is a regular dictionary with an extra attribute-style access, meaning I can now write it all like so:
from bunch import Bunch MESSAGE = Bunch() MESSAGE.CREATE_FOO = '1000' MESSAGE.DELETE_FOO = '1001' # And so on..
# Yay, we're importing one thing only, the only one needed # in this module, and if the code below ever starts deleting # FOO, we won't have to update the import statement! from constants import MESSAGE print(MESSAGE.CREATE_FOO)
Isn’t that nicer?
Here’s another one in the hoping-it-saves-someone-at-least-half-an-hour-worth-of-searching-around category which shows nothing but how to define an ON DELETE CASCADE constraint and then how to issue a LEFT JOIN in SQLAlchemy.
# -*- coding: utf-8 -*- # SQLAlchemy from sqlalchemy import create_engine from sqlalchemy import Column, Integer, String, ForeignKey from sqlalchemy.ext.declarative import declarative_base from sqlalchemy.orm import backref, relationship, sessionmaker Base = declarative_base() # The definitions, note the use of both 'ondelete' and 'cascade'. class Mom(Base): __tablename__ = 'mom' id = Column(Integer, primary_key=True) name = Column(String(60)) class Daughter(Base): __tablename__ = 'daughter' id = Column(Integer, primary_key=True) name = Column(String(60)) mom_id = Column(Integer, ForeignKey('mom.id', ondelete='CASCADE'), nullable=False) mom = relationship(Mom, backref=backref('daughters', cascade='all, delete, delete-orphan')) class Son(Base): __tablename__ = 'son' id = Column(Integer, primary_key=True) name = Column(String(60)) mom_id = Column(Integer, ForeignKey('mom.id', ondelete='CASCADE'), nullable=False) mom = relationship(Mom, backref=backref('sons', cascade='all, delete, delete-orphan')) # Create an in-memory database. engine = create_engine('sqlite:///:memory:', echo=True) Base.metadata.create_all(engine) Session = sessionmaker(bind=engine) session = Session() # Let's find out how LEFT JOIN works, note the usage of .outerjoin mom1 = Mom() mom1.name = 'Molly' mom2 = Mom() mom2.name = 'Sarah' mom3 = Mom() mom3.name = 'Martha' daughter = Daughter() daughter.name = 'Matilda' daughter.mom = mom1 son1 = Son() son1.name = 'Robert' son1.mom = mom1 son2 = Son() son2.name = 'Tom' son2.mom = mom2 session.add_all([mom1, mom2, mom3]) session.commit() session.delete(mom2) session.commit() # All moms and their children. rows1 = session.query(Mom.id, Mom.name, Daughter.name.label('daughter_name'), Son.name.label('son_name')).\ outerjoin(Daughter, Mom.id==Daughter.mom_id).\ outerjoin(Son, Mom.id==Son.mom_id).\ order_by('mom.name').\ all() # Sarah doesn't like us anymore so Tom should be gone as well. rows2 = session.query(Son.id, Son.name).all() for row in rows1: print(row) print('') for row in rows2: print(row)
As expected, the result is
(3, u'Martha', None, None) (1, u'Molly', u'Matilda', u'Robert') (1, u'Robert')
We can also confirm the SQL code that’s being generated – notice the ON DELETE CASCADE clause ..
CREATE TABLE mom ( id INTEGER NOT NULL, name VARCHAR(60), PRIMARY KEY (id) ) CREATE TABLE daughter ( id INTEGER NOT NULL, name VARCHAR(60), mom_id INTEGER NOT NULL, PRIMARY KEY (id), FOREIGN KEY(mom_id) REFERENCES mom (id) ON DELETE CASCADE ) CREATE TABLE son ( id INTEGER NOT NULL, name VARCHAR(60), mom_id INTEGER NOT NULL, PRIMARY KEY (id), FOREIGN KEY(mom_id) REFERENCES mom (id) ON DELETE CASCADE )
.. and the LEFT JOIN is here indeed.
SELECT mom.id AS mom_id, mom.name AS mom_name, daughter.name AS daughter_name, son.name AS son_name FROM mom LEFT OUTER JOIN daughter ON mom.id = daughter.mom_id LEFT OUTER JOIN son ON mom.id = son.mom_id ORDER BY mom.name
I had to implement a couple of things related to the validation of client SSL/TLS certificates – as returned by the ssl.SSLSocket.getpeercert(True) method – and I thought I’d share the code with others, I guess there’s never too much of sample code in that area. Turned out that everything went smoothly with ssl and M2Crypto and it’s all below. Note that the code downloads the certificate off the Internet but it will work properly with a client certificate as well – well, maybe except for the fact that getpeercert(True) returns a DER while get_server_certificate gives you a PEM – but otherwise it will work just fine. Cheers!
# -*- coding: utf-8 -*- from __future__ import absolute_import, division, print_function, unicode_literals # stdlib import ssl # M2Crypto from M2Crypto import X509 # Grab a certificate from the nearest server .. cert_pem = ssl.get_server_certificate(('duckduckgo.com', 443)) # .. let's start with converting PEM to DER .. cert_der = ssl.PEM_cert_to_DER_cert(cert_pem) # .. how does the DER look like (not too pretty).. print('\nDER:') print(cert_der) # .. M2Crypto has a bunch of useful functions .. x509 = X509.load_cert_string(cert_pem, X509.FORMAT_PEM) # .. does the certificate belong to a CA .. print('\nIs it a CA:') print(x509.check_ca()) # .. the fingerprint .. fp = x509.get_fingerprint('sha1') fp = ':'.join(fp[pos:pos+2] for pos in xrange(0, len(fp), 2)) print('\nSHA1 fingerprint:') print(fp) # .. serial number .. print('\nSerial number:') print(x509.get_serial_number()) # .. who has it been issued by .. print('\nIssuer:') print(x509.get_issuer()) # .. how about the subject's commonName and organization only .. subject = x509.get_subject() print('\nSubject\'s commonName and organization:') print(subject.CN) print(subject.O) # .. public key's exponent .. # inspired by http://mail.python.org/pipermail/python-crypto/2004-August/000683.html e_data = x509.get_pubkey().get_rsa().e # First 4 bytes are the length of what then follows length, e_raw = e_data[:4], e_data[4:] e = 0 for c in e_raw: e = (e*256) + ord(c) print('\nPublic key\'s exponent:') print(e)
HAProxy exposes some if its statistics through a UNIX socket and it turns out it’s very easy to access it from Python, like below.
I guess the code is pretty self-explanatory – essentially it’s expecting you to give it a path to the socket and then to feed it with commands. Some commands may need an extra argument, like ’show sess’ does and if you’re particularly impatient – or maybe you’re under a very heavy load – you can also you use the ‘timeout’ parameter to specify how long you’re willing to wait for HAProxy to reply.
The responses are returned as-is and actually I wouldn’t mind it if someone took the code and added a nice OO look’n'feel to it so that it read like here ..
stats = HAProxyStats('/tmp/haproxy-stat.sock') sess = stats.sess('0x19e8fd0') print(sess.svname)
.. that could make for a useful project but for now, here’s some code that works today
# -*- coding: utf-8 -*- from __future__ import absolute_import, division, print_function, unicode_literals # stdlib import logging import socket from cStringIO import StringIO from time import time from traceback import format_exc logger = logging.getLogger(__name__) class HAProxyStats(object): """ Used for communicating with HAProxy through its local UNIX socket interface. """ def __init__(self, socket_name=None): self.socket_name = socket_name def execute(self, command, extra="", timeout=200): """ Executes a HAProxy command by sending a message to a HAProxy's local UNIX socket and waiting up to 'timeout' milliseconds for the response. """ if extra: command = command + ' ' + extra buff = StringIO() end = time() + timeout client = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) try: client.connect(self.socket_name) client.send(command + '\n') while time() <= end: data = client.recv(4096) if data: buff.write(data) else: return buff.getvalue() except Exception, e: msg = 'An error has occurred, e=[{e}]'.format(e=format_exc(e)) logger.error(msg) raise finally: client.close() if __name__ == '__main__': stats = HAProxyStats('/tmp/haproxy-stat.sock') info = stats.execute('show info') print(info) stat = stats.execute('show stat') print(stat) sess = stats.execute('show sess', '0x19e8fd0', 20) print(sess)
I’ve put a new PyYAML mirror online at http://pyyaml-mirror.gefira.pl/simple/ so that it can be used in emergency cases, like now, when the main site http://www.pyyaml.org has apparently been down for some time. Hope it saves someone at least a little bit of time!
I’ve been looking for information on how to get rid of the ‘No Cython, trying Pyrex…’ warning popping up when using things like bzr in zc.builduot config files but there was no clear answer to it so what I’ve finally come up with is below. Essentially, Cython is first downloaded and then compiled so that bzr – or anything that will benefit from there being Cython installed – can freely make use of it.
It isn’t particularly beautiful but does the trick at least, hoping it will save time for someone else!
[buildout] parts = cython-src cython-install myapp [config] cython_version = 0.14 [myapp] recipe = zc.recipe.egg interpreter = py eggs = bzr [cython-src] recipe = hexagonit.recipe.download url = http://pypi.python.org/packages/source/C/Cython/Cython-${config:cython_version}.tar.gz [cython-install] recipe = iw.recipe.cmd on_install = true cmds = cd ${buildout:directory}/parts/cython-src/Cython-${config:cython_version}; ../../../bin/python setup.py install
The unstoppable Miguel Landaeta did it again and I’m happy to let you all know that sec-wall, the security proxy, can be now installed on Debian wheezy/sid using nothing but DEBs, like below. The only prerequisite is that you need to visit http://alioth.debian.org/~nomadium-guest/debian/unstable/ to find and download the latest DEB – at the time of this writing it’s sec-wall_1.0.0-0miguel2_all.deb but it sure is going to change shortly because sec-wall 1.1 is about to be released soon.
First, the dependencies:
apt-get install python-springpython python-gevent apt-get install python-argparse python-lxml apt-get install python-pesto python-zdaemon apt-get install python-pkg-resources
Now install sec-wall:
dpkg -i ./sec-wall_1.0.0-0miguel2_all.deb
And that’s all!
Nice folks at www.linuxsecurity.com have just published a new article on using the sec-wall security proxy with cURL and PycURL, the Python interface to the libcurl library, so if you’re looking for information on how to test services secured using sec-wall, either with HTTP Basic/Digest Auth, custom HTTP headers, XPath-based authentication, WS-Security or SSL/TLS client certificates then you won’t be disappointed. Enjoy!
One of the things sec-wall, a featured-packed high performance security proxy, provides is the support for securing access to resources using arbitrary XPath expressions. What is currently missing in the documentation though is an explanation of how one should use XML namespaces. The thing can be done and there’s a bug report regarding it which I’m going to fix and close in a day or two but just thought that in the meantime I’d blog about it.
So how would one go about creating a sec-wall config.py file that should let in only clients that use credentials akin to what’s below?
<?xml version="1.0" encoding="utf-8"?> <a> <b> <username xmlns="http://example.com/myns1">foo</username> <c xmlns="http://example.com/myns2" password="bar" /> </b> </a>
The answer is pretty simple – etree.XPath objects accept a namespaces argument which ought to be a mapping between prefixes used in expressions and actual namespaces, so the config file should read like below:
# -*- coding: utf-8 -*- # stdlib import uuid # lxml from lxml import etree # Don't share it with anyone. INSTANCE_SECRET = '7bcb90942d994440af05d02b691ae86d' # May be shared with the outside world. INSTANCE_UNIQUE = uuid.uuid4().hex # ############################################################################## def xpath(): username = 'foo' password = 'bar' xpath1 = "/a/b/myns1:username/text() = '{0}'".format(username) xpath2 = "//myns2:c/@password='{0}'".format(password) ns_dict = { 'myns1': 'http://example.com/myns1', 'myns2': 'http://example.com/myns2', } return { 'xpath': True, 'xpath-1': etree.XPath(xpath1, namespaces=ns_dict), 'xpath-2': etree.XPath(xpath2, namespaces=ns_dict), 'host': 'http://example.com/', } urls = [ ('/xpath', xpath()), ]
Solid, eh?
