gefira blog

Nice folks at www.linuxsecurity.com have just published a new article on using the sec-wall security proxy with cURL and PycURL, the Python interface to the libcurl library, so if you’re looking for information on how to test services secured using sec-wall, either with HTTP Basic/Digest Auth, custom HTTP headers, XPath-based authentication, WS-Security or SSL/TLS client certificates then you won’t be disappointed. Enjoy!

One of the things sec-wall, a featured-packed high performance security proxy, provides is the support for securing access to resources using arbitrary XPath expressions. What is currently missing in the documentation though is an explanation of how one should use XML namespaces. The thing can be done and there’s a bug report regarding it which I’m going to fix and close in a day or two but just thought that in the meantime I’d blog about it.

So how would one go about creating a sec-wall config.py file that should let in only clients that use credentials akin to what’s below?

<?xml version="1.0" encoding="utf-8"?>
<a>
    <b>
        <username xmlns="http://example.com/myns1">foo</username>
        <c xmlns="http://example.com/myns2" password="bar" />
    </b>
</a>

The answer is pretty simple – etree.XPath objects accept a namespaces argument which ought to be a mapping between prefixes used in expressions and actual namespaces, so the config file should read like below:

# -*- coding: utf-8 -*-
 
# stdlib
import uuid
 
# lxml
from lxml import etree
 
# Don't share it with anyone.
INSTANCE_SECRET = '7bcb90942d994440af05d02b691ae86d'
 
# May be shared with the outside world.
INSTANCE_UNIQUE = uuid.uuid4().hex
 
# ##############################################################################
 
def xpath():
 
    username = 'foo'
    password = 'bar'
 
    xpath1 = "/a/b/myns1:username/text() = '{0}'".format(username)
    xpath2 = "//myns2:c/@password='{0}'".format(password)
 
    ns_dict = {
        'myns1': 'http://example.com/myns1',
        'myns2': 'http://example.com/myns2',
    }
 
    return {
        'xpath': True,
        'xpath-1': etree.XPath(xpath1, namespaces=ns_dict),
        'xpath-2': etree.XPath(xpath2, namespaces=ns_dict),
        'host': 'http://example.com/',
    }
 
urls = [
    ('/xpath', xpath()),
]

Solid, eh?

I’ve just pushed a new feature to the lp:~pymqi-dev/pymqi/string-filter branch that makes it possible to use WebSphere MQ PCF/MQAI filters with Python. That should be a nice addition if anyone’s thinking of creating cross-platform open-source equivalents of various Windows-only freeware support packs there are already available. Or just for anyone writing slightly more advanced code for administering MQ objects.

So how does it work?

You sure remember how to use PCFExecute, right? Well, the class has just been extended and now accepts a new parameter, a list of filters to use when querying MQ objects. The API has been more or less inspired by how Django’s QuerySets look like which basically means it’s a higher level one then what you’d be normally using with plain MQ verbs. See for yourself, the code below fetches all the queues that have a description starting with a ‘WebSphere MQ Selection*’ pattern and the current depth of 10 and greater:

import pymqi, CMQC, CMQCFC
 
queue_manager = 'QM01'
channel = 'SVRCONN.1'
host = '192.168.1.139'
port = '1434'
conn_info = '%s(%s)' % (host, port)
 
qmgr = pymqi.connect(queue_manager, channel, conn_info)
pcf = pymqi.PCFExecute(qmgr)
 
attrs = {CMQC.MQCA_Q_NAME :'*', CMQC.MQIA_Q_TYPE : CMQC.MQQT_LOCAL}
 
desc_filter = pymqi.Filter(CMQC.MQCA_Q_DESC).like('WebSphere MQ Selection*')
depth_filter = pymqi.Filter(CMQC.MQIA_CURRENT_Q_DEPTH).greater(10)
 
res = pcf.MQCMD_INQUIRE_Q(attrs, [desc_filter, depth_filter])
print(res)

It’s still a feature branch not trunk, and even if it were trunk it still hasn’t been released in PyMQI 1.3, and even then, there still would be PyMQI 1.4, so if you feel like anything should be changed or rather work differently, please speak up! Cheers!

One of the niceties of sec-wall, the security proxy, is that it’s written in Python which means not only that one doesn’t have to learn yet another obscure configuration format summoned from the deepest levels of R’lyeh but also that it’s very easy to extend the config.py file.

For instance, by default all of the secrets need to be given in clear text and that can probably be sub-optimal for some people, so why don’t we customize the config file and have it use GnuPG with symmetric decryption via the python-gnupg wrapper?

Let’s install python-gnupg first (piece of cake):

$ sudo pip install python-gnupg

Let’s prepare the secret.ini file, an INI-formatted configuration file which contains all the secrets:

$ cat secret.ini
[secret]
INSTANCE_SECRET=756817c1e2b8443c9d6f728da9b2b5fd
admin=MySecretPassword
$

The file will be now encrypted using GnuPG (let’s say the password is ’123′) ..

$ gpg --symmetric secret.ini
$

.. the result of which is a new secret.ini.gpg file and secret.ini may now be removed:

$ rm secret.ini
$

OK, we need the accompanying config.py file now, just like the one below:

# -*- coding: utf-8 -*-
 
# stdlib
import cStringIO, getpass, sys, uuid
from ConfigParser import ConfigParser
 
# python-gnupg
import gnupg
 
# Where are we expecting the secrets to be.
gpg_file = open('./secret.ini.gpg', "rb")
 
passphrase = getpass.getpass('Enter passphrase (will not be echoed back): ')
 
gpg = gnupg.GPG()
ini_data = str(gpg.decrypt_file(gpg_file, passphrase=passphrase))
 
# Exit early on an incorrect passphrase.
if not ini_data:
    print("Incorrect passphrase.")
    sys.exit(1)
 
# The passphrase was OK, proceed on to parsing the INI data.
config = ConfigParser()
config.readfp(cStringIO.StringIO(ini_data))
 
# Use the GPG-encrypted data.
INSTANCE_SECRET = config.get('secret', 'INSTANCE_SECRET')
 
# May be shared with the outside world.
INSTANCE_UNIQUE = uuid.uuid4().hex
 
# ##############################################################################
 
def admin():
    return {
        'basic-auth': True,
        'basic-auth-username':'user',
        'basic-auth-password':config.get('secret', 'admin'), # Uses GPG now.
        'basic-auth-realm':'Secure area',
        'host': 'http://example.com'
    }
 
def default():
    return {
        'ssl': True,
        'ssl-cert': True,
        'ssl-cert-commonName':INSTANCE_SECRET,
        'host': 'http://' + INSTANCE_SECRET
    }
 
urls = [
    ('/admin/', admin()),
    ('/*', default()),
]

Try starting the proxy now. If everything’s OK, if the passphrase you’ve supplied has been the same as the one used for encrypting the original INI file, the sec-wall instance will start in background as usual:

$ sec-wall --start .
Enter passphrase (will not be echoed back):
$ echo $?
0
$

If on the other hand the password wasn’t ’123′, the proxy won’t start:

$ sec-wall --start .
Enter passphrase (will not be echoed back):
Incorrect passphrase.
$ echo $?
1
$

And that’s it! Couldn’t have been simpler, eh?

This is mostly a copy of the announcement sent to the python-announce@ list, although I’ve added a couple more usage examples at the end of the post.

sec-wall is a feature packed high-performance security proxy which has many interesting features, including the support for SSL/TLS, WS-Security, HTTP Auth Basic/Digest, extensible authentication schemes based on custom HTTP headers and XPath expressions, powerful URL matching/rewriting and an optional headers enrichment.

sec-wall uses and is built on top of several fantastic Python open source technologies, such as gevent, Spring Python, Pesto, lxml, zdaemon or PyYAML and is meant to be highly customizable and easy to use. Good performance, tests, documentation and building an awesome community are at the very heart of the project.

Here are  examples showing how little is needed to secure a backend server with HTTP Basic Auth, SSL/TLS client certificates and WS-Security.

# -*- coding: utf-8 -*-
 
# stdlib
import uuid
 
# Don't share it with anyone.
INSTANCE_SECRET = '5bf4e78c256746eda2ce3e0e73f256d0'
 
# May be shared with the outside world.
INSTANCE_UNIQUE = uuid.uuid4().hex
 
def default():
    return {
        'basic-auth':True,
        'basic-auth-username':'MyUser',
        'basic-auth-password':'MySecret',
        'basic-auth-realm':'Secure area',
        'host': 'http://example.com'
    }
 
urls = [
    ('/*', default()),
]

# -*- coding: utf-8 -*-
 
# stdlib
import os.path as path, uuid
 
# Don't share it with anyone.
INSTANCE_SECRET = '5bf4e78c256746eda2ce3e0e73f256d0'
 
# May be shared with the outside world.
INSTANCE_UNIQUE = uuid.uuid4().hex
 
# Useful constants
cur_dir = path.dirname(__file__)
 
# Crypto
keyfile = path.join(cur_dir, './crypto/server-priv.pem')
certfile = path.join(cur_dir, './crypto/server-cert.pem')
ca_certs = path.join(cur_dir, './crypto/ca-cert.pem')
 
server_type = 'https'
 
def default():
    return {
        'ssl': True,
        'ssl-cert': True,
        'ssl-cert-commonName': 'My Client',
        'ssl-cert-organizationName': 'My Company',
        'host': 'http://example.com'
    }
 
urls = [
    ('/*', default()),
]

# -*- coding: utf-8 -*-
 
# stdlib
import uuid
 
# Don't share it with anyone.
INSTANCE_SECRET = '5bf4e78c256746eda2ce3e0e73f256d0'
 
# May be shared with the outside world.
INSTANCE_UNIQUE = uuid.uuid4().hex
 
def default():
    return {
        'wsse-pwd': True,
        'wsse-pwd-username': 'MyUser',
        'wsse-pwd-password': 'MySecret', # Needs to be given in clear text
        'wsse-pwd-reject-empty-nonce-creation': True,
        'wsse-pwd-reject-stale-tokens': True,
        'wsse-pwd-reject-expiry-limit': 120,
        'wsse-pwd-nonce-freshness-time': 120,
        'host': 'http://example.com'
    }
 
urls = [
    ('/*', default()),
]

Links:

Project’s homepage: http://sec-wall.gefira.pl/
Getting started: http://sec-wall.gefira.pl/documentation/getting-started/index.html
Usage examples: http://sec-wall.gefira.pl/documentation/usage-examples/index.html
Twitter: https://twitter.com/fourthrealm
Blog: http://www.gefira.pl/blog
IRC: #sec-wall channel on Freenode network

Cheers!

Woo hoo! PyMQI 1.2 – Python interface to WebSphere MQ – has been released and here’s an excerpt on new features taken from the release announcement:

• Added support for MQ 7.0-style publish/subscribe
• Added support for creating and parsing of MQRFH2 headers
• Added new constants in the CMQZC.py module
• Added new structures – MQSRO, MQSD, MQTM, MQTMC2
• PyMQI now supports byte strings
• New MQ verb – MQSUB
• Simplified establishing connections to queue managers
• Added means for checking whether a client application is connected to a queue manager

Special thanks to Hannes Wagener for his excellent contributions!

Links

Project’s homepage: http://packages.python.org/pymqi/
Download URL: https://launchpad.net/pymqi/+download
Usage examples: http://packages.python.org/pymqi/examples.html
Twitter: https://twitter.com/fourthrealm
Blog: http://www.gefira.pl/blog
LinkedIn: http://www.linkedin.com/groups?gid=3726448
IRC: #pymqi channel on Freenode network

Hi and welcome to the first installment in the WebSphere MQ Administration with Python series. The goal is to introduce readers to the topic of performing admin tasks with Python and encourage those who still waver over whether to migrate all those Visual Basic or Java tools to actually start using Python, a modern, fast and cross-platform programming language.

The Python interface to MQ is called PyMQI and among many goodies it provides there’s also a very useful pymqi.PCFExecute class. PCF is, as you surely remember, a short for Programmable Command Formats, which basically means it’s an API for writing programs that are meant to automate tasks MQ admins usually do.

How does it work in Python? Well, you just need to open the appropriate MQ manual and pick any PCF command, say an MQCMD_PING_Q_MGR. The name you’ve chosen is always available as an attribute of the PCFExecute class instances, like below:

import pymqi
 
queue_manager = "QM01"
channel = "SVRCONN.1"
host = "192.168.1.139"
port = "1434"
conn_info = "%s(%s)" % (host, port)
 
qmgr = pymqi.QueueManager(None)
qmgr.connectTCPClient(queue_manager, pymqi.cd(), channel, conn_info)
pcf = pymqi.PCFExecute(qmgr)
 
# Woohoo, we've just executed our first PCF command!
pcf.MQCMD_PING_Q_MGR()
 
qmgr.disconnect()

So you just need to connect to a queue manager, create an instance of the pymqi.PCFExecute class and then you can directly invoke anything that’s listed in MQ manuals.

OK, that was easy as pinging a queue manager doesn’t require any parameters nor does it return anything. How about something less trivial, like creating a new queue with MQCMD_CREATE_Q?

import CMQC
import pymqi
 
queue_manager = "QM01"
channel = "SVRCONN.1"
host = "192.168.1.139"
port = "1434"
conn_info = "%s(%s)" % (host, port)
 
qmgr = pymqi.QueueManager(None)
qmgr.connectTCPClient(queue_manager, pymqi.cd(), channel, conn_info)
pcf = pymqi.PCFExecute(qmgr)
 
queue_name = "PYMQI.1"
queue_type = CMQC.MQQT_LOCAL
 
args = {CMQC.MQCA_Q_NAME: queue_name, CMQC.MQIA_Q_TYPE: queue_type}
 
pcf.MQCMD_CREATE_Q(args)
 
qmgr.disconnect()

MQCMD_CREATE_Q obviously needs some arguments and these are passed in wrapped in a dictionary whose keys are the same as the manual says there should be. For instance, it says that MQCA_Q_NAME and MQIA_Q_TYPE are required identifiers so I knew I could use the PyMQI’s CMQC module for looking them up. Likewise, the manual lists all the supported queue types and they also can be found in CMQC and that’s how I knew I could use CMQC.MQQT_LOCAL for the queue type. So to recap, the arguments – required and optional – are explained in the MQ documentation and you need to use PyMQI constants to form a dictionary and pass it in to the PCF command of your choice. What other modules are there in addition to CMQC? Well, there’s also CMQXC and CMQCFC and PyMQI 1.2 will introduce CMQZC – they all may be used for constructing the arguments to PCF calls.

So here it is, an introduction to PCFExecute. I plan on writing more about manipulating MQ objects, queues, channels, queue managers, aliases and so on and if you’d like to have anything covered you just need to drop me an e-mail. Thanks!

Some time ago I was in a middle of resolving an interesting issue in PyMQI while I found myself in a need of a clear example of how to use WebSphere MQ’s MQSUBRQ in C. Yea, in C this time, not in Python. I needed something that wouldn’t have been obscured by stuff like parsing of command line options or you know, anything that painfully reminds you how low-level C really is.

So I wrote the code and thought I shouldn’t be really keeping it for myself, maybe someone else will need it some day as well. Hence I’ve uploaded it all to my bitbucket repo, and if you need the sample code for MQCONNX, MQPUT, MQSUB, MQSUBRQ and then, finally, an MQGET then here it is, enjoy!

And while I have your attention – you do know there’s PyMQI, the Python interface to WebSphere MQ, don’t you? So if one shiny beautiful day you find yourself weary of C then you are more than welcome to give PyMQI and Python a try!

I’ve noticed quite a few people hitting this blog and looking for keywords along the lines of “pymqi read a message and leave on queue” so I thought it should be explained. The thing you’re looking for is called message browsing and this post will show it in action.

Before I get into details though, I’d like to ask you for contacting me if there’s anything I can do regarding the documentation and usage examples. Really, it’s open-source so both the code and docs should shine but I’ll never know what to improve next if you don’t tell me it first Checking the referrers is of course important but it won’t help much if you shun sending me your suggestions. Thanks!

So, the code is below.  The trick is composed of two parts that sort of resemble using cursors in SQL databases:

• The queue needs to be open for browsing using the CMQC.MQOO_BROWSE option,
• When calling queue.get, you need to pass it a proper CMQC.MQGMO_BROWSE_* option, in our simple case, there’s only one message on the queue and it just so happens that right after opening the queue for browsing, the logical cursor used for browsing is placed before the first message so we can use CMQC.MQGMO_BROWSE_NEXT and it will browse the next message which happens to be the first and the only one on the queue.

The code has been written in a form of a test case in the hope that it will make it clear that msg1 and msg2 are really equal to self.expected_msg, which is a UUID4 so we know it’s a different message on each test invocation.

I think that would be it as far as explaining the concept goes and again, don’t hesitate to contact me if you’d like me to expand on it or maybe cover something entirely else.

# stdlib
import unittest, uuid
 
# PyMQI
import CMQC, pymqi
 
queue_manager = "QM01"
channel = "SVRCONN.1"
host = "192.168.1.139"
port = "1434"
queue_name = "TEST.1"
conn_info = "%s(%s)" % (host, port)
 
class BrowsingTestCase(unittest.TestCase):
    def setUp(self):
        self.expected_msg = uuid.uuid4().hex
        self.qmgr = pymqi.QueueManager(None)
        self.qmgr.connectTCPClient(queue_manager, pymqi.cd(), channel, conn_info)
 
        self.put_queue = pymqi.Queue(self.qmgr, queue_name)
        self.put_queue.put(self.expected_msg)
        self.put_queue.close()
 
        browse_options = CMQC.MQOO_BROWSE
        self.browse_queue = pymqi.Queue(self.qmgr, queue_name,  browse_options)
        self.get_queue = pymqi.Queue(self.qmgr, queue_name)
 
    def tearDown(self):
        # Clear the queue and close all objects.
        try:
            while True:
                self.get_queue.get()
        except pymqi.MQMIError, e:
            if e.comp != CMQC.MQCC_FAILED and e.reason != CMQC.MQRC_NO_MSG_AVAILABLE:
                raise
 
        self.browse_queue.close()
        self.get_queue.close()
        self.qmgr.disconnect()
 
    def test_message_browsing(self):
 
        md = pymqi.md()
        gmo = pymqi.gmo()
        gmo.Options = CMQC.MQGMO_BROWSE_NEXT
 
        msg1 = self.browse_queue.get(None, md, gmo)
        self.assertEquals(msg1, self.expected_msg)
 
        msg2 = self.get_queue.get()
        self.assertEquals(msg2, self.expected_msg)

Spring Python 1.2 has been released some time ago and one of its newest features is the support for wrapping XML-RPC with SSL/TLS, including both clients and servers. I’ve covered it already in a tutorial some time ago but I don’t think Planet Python was picking up the blog’s posts at that time, besides, Spring Python was still a release candidate when I was writing it so things could’ve changed (but they didn’t ). In any case, now that 1.2 is a stable release, I thought I’d just mention it in case you were looking for means to secure your XML-RPC traffic. Here’s the reference material and the tutorial is here.