ralph-heinkel.com (Posts about Python)

EuroPython 2014 pyRserve-slides now available online

Ralph Heinkel — Fri, 19 Sep 2014 09:50:00 GMT

After our well-attended Python Meetup last Wednesday in Heidelberg about "Connecting Python to other programming languages for scientific computing" I had been approached to publish the slides of my pyRserve talk (which has been the same as the one I've given at EuroPython in Berlin, for those who are interested). These are now online, so feel free to download them.

pyRserve 0.8.1 released

Ralph Heinkel — Fri, 18 Jul 2014 08:20:00 GMT

About pyRserve

pyRserve is a (pure python) client for connecting Python to an R process on a remote server via TCP-IP (using Rserve). R is one of the most important and most widely spread open source statistic packages available.

Through such a pyRserve connection the full power of R is available on the Python side without programming in R. From Python variables can get set in and retrieved from R, and R-functions can be created and called remotely. No R-related libraries need to be installed on the client side, pip install pyRserve is all that needs to be done.

Sample code

This code assumes that Rserve (the part that connects the R engine to the network) already is running. Details can be found in the pyRserve docs.

>>> import pyRserve
>>> conn = pyRserve.connect('Rserve.domain.com')
>>> conn.eval('sum( c(1.5, 4) )') # direct evaluation of a statement in R
5.5
>>> conn.r.myList = [1, 2, 3] # bind a Python list in R to variable 'myList'

>>> conn.voidEval('func1 <- function(v) { v*2 }')  # create a function in R
>>> conn.r.func1(4)                                # call the function in R
16

Most important changes in V 0.8.x

Support for out-of-bound messages (allows for callbacks from R to Python) (contrib. by Philip. A.)
Rserve can now be shutdown remotely (contrib. by Uwe Schmitt)
Fixed bug when passing R functions as parameters to R functions
Documentation errors have been fixed

Documentation and Support

The documentation for pyRserve is available at http://packages.python.org/pyRserve
The corresponding Google news group can be found at http://groups.google.com/group/pyrserve

Publish-Subscribe with web sockets in Python and Firefox

Ralph Heinkel — Sun, 22 Jul 2012 15:20:00 GMT

WebSockets provide a way to communicate through a bi-directional channel on a single TCP connection. This technology is especially interesting since it allows a web server to push data to a browser (client) without having the client to constantly poll for it. In contrast to normal HTTP requests where a new TCP connection gets opened and closed for each request web socket connections are kept open until one party closes them. This allows for communication in both directions, and calls can be made multiple times on the same connection.

In this little article I basically combine what I found on Sylvain Hellegouarch's documentation for ws4py (a WebSocket client and server library for Python) and the article HTML5 Web Socket in Essence by Wayne Ye.

More specifically the examples below shows how multiple clients subscribe via websockets to a cherrypy server through a web socket connection. The first of the two clients in the example below is a very lightweight client based solely on the ws4py package, the other (javascript) implementation is supposed to run in Firefox.

The server

This example provides a minimal publishing engine implemented with cherrypy. An instance of class WebSocketTool is hooked up into cherrypy as a so-called cherrypy tool, and a web socket handler (the Publisher-class) is bound to this tool as a handler for calls to the path /ws:

import cherrypy
from ws4py.server.cherrypyserver import WebSocketPlugin, WebSocketTool
from ws4py.websocket import WebSocket

cherrypy.config.update({'server.socket_port': 9000})
WebSocketPlugin(cherrypy.engine).subscribe()
cherrypy.tools.websocket = WebSocketTool()

SUBSCRIBERS = set()

class Publisher(WebSocket):
    def __init__(self, *args, **kw):
        WebSocket.__init__(self, *args, **kw)
        SUBSCRIBERS.add(self)

    def closed(self, code, reason=None):
        SUBSCRIBERS.remove(self)

class Root(object):
    @cherrypy.expose
    def index(self):
        return open('ws_browser.html').read()

    @cherrypy.expose
    def ws(self):
        "Method must exist to serve as a exposed hook for the websocket"

    @cherrypy.expose
    def notify(self, msg):
        for conn in SUBSCRIBERS:
            conn.send(msg)

cherrypy.quickstart(Root(), '/', 
    config={'/ws': {'tools.websocket.on': True,
                    'tools.websocket.handler_cls': Publisher}})

The only purpose of the Root.ws()-method is to make this method available under /ws in the web server through the cherrypy.expose decorator. Whenever a websocket client makes a request to /ws an instance of class Publisher is created, which registers itself to the global SUBSCRIBERS set on __init__(). When the server goes down, or the client disconnects, its closed() method is called.

The only packages needed for this example are cherrypy and ws4py. Both can be easily installed via easy_install or pip. Save the code above as ws_server.py and start it with

python ws_server.py

Now the server is ready to accept client connections through the web socket protocol. As soon as one of the clients described below has subscribed to this server messages can be published by calling the Root.notify() method. Since it is exposed it is possible to call it from the command line with

curl localhost:9000/notify?msg=HelloWorld

Of course wget works as well.

A pure Python client

The Python client's code is quite short. ws4py provides three sample client implementations, the threaded one has been chosen for this example. The others are using gevent or Tornado.

from ws4py.client.threadedclient import WebSocketClient

class Subscriber(WebSocketClient):
    def handshake_ok(self):
        self._th.start()
        self._th.join()

    def received_message(self, m):
        print "=> %d %s" % (len(m), str(m))

if __name__ == '__main__':
    ws = Subscriber('ws://localhost:9000/ws')
    ws.connect()

The method handshake_ok() has been overridden to keep the thread stored in self._th running continuously (the original implementation quits after one second). After the Subscriber-class has been instantiated it connects to the cherrypy server. Whenever the server sends a message it will be delegated to the method received_message() where it gets printed to stdout.

Just store this code into a file, e.g. ws_subscriber.py and start it in from a new shell. The cherrypy server should print a message to the console that it received a web socket connection.

Now again call the notify-method in the server:

curl localhost:9000/notify?msg=HelloWorld

and the python client should print your message to the screen.

A web socket client in Firefox

This browser client uses the web socket protocol built into Firefox. The example below works for me in FF14, it failed for FF8. I'm not sure which version of Firefox starts to support it. Safari version 5.0 also fails. IE has not been tested.

<html>
  <head>
    <script>
      var websocket = new WebSocket('ws://localhost:9000/ws');
      websocket.onopen    = function (evt) { console.log("Connected to WebSocket server."); };
      websocket.onclose   = function (evt) { console.log("Disconnected"); };
      websocket.onmessage = function (evt) { document.getElementById('msg').innerHTML = evt.data; };
      websocket.onerror   = function (evt) { console.log('Error occured: ' + evt.data); };
    </script>
  </head>
  <body>
    <h1>Websocket demo</h1>
    Message: <span id="msg" />
  </body>
</html>

At load time a Websocket-instance is created and event handlers are installed. The interesting one is the onMessage-handler: it is called for each message received, it copies the message into the SPAN element and thus makes it visible.

Make sure to store this html page in the same directory as the ws_server.py above since we are going to open it through cherrypy's index method. For this to work it has to be named ws_browser.html. Now open Firefox and direct it to http://localhost:9000. You should immediately see this page. The SPAN element should be empty.

Again repeat the curl or wget command in your shell and both the python client (if it is still running) and the SPAN element should display your "HelloWorld" message.

pyRserve 0.6.0 released

Ralph Heinkel — Wed, 04 Jul 2012 17:20:00 GMT

While being at EuroPython in Florence the latest version of pyRserve has now been finished and is available via pypi (easy_install -U pyRserve). If you are at EuroPython, too, and want to talk about it just come and see me.

About pyRserve

Through such a pyRserve connection the full power of R is available on the Python side without programming in R. From Python variables can get set in and retrieved from R, and R-functions can be called remotely. No R-related libraries need to be installed on the client side.

Sample code

>>> import pyRserve
>>> conn = pyRserve.connect('servername.domain.com')
>>> conn.r('1+1')                # direct evaluation of a statement in R
2
>>> conn.r.myList = [1, 2, 3]    # bind a Python list in R to variable 'myList'

>>> conn.r('func1 <- function(v) { v*2 }') # create a function in R
>>> conn.r.func1(4)                        # call the function in R
16

Most important changes in V 0.6.0

Support for Python 3.x (therefore dropped support for Python <= 2.5)
Support for unicode strings
Suport for Fortran-style ordering of numpy arrays
Elements of single-item arrays are now translated to native python data types
Full support complex numbers, partial support for 64bit integers and arrays

Documentation and Support

The documentation for pyRserve is available at http://packages.python.org/pyRserve
The corresponding Google news group can be found at http://groups.google.com/group/pyrserve

pyRserve 0.5.2 released

Ralph Heinkel — Tue, 06 Dec 2011 18:22:00 GMT

The latest version is now available via pypi (easy_install -U pyRserve).

About pyRserve

pyRserve is a (pure python) client for connecting Python to an R process on a remote server via TCP-IP (using Rserve). Through such a connection variables can be get and set in R from Python, and also R-functions can be called remotely. No R-related libraries need to be installed on the client side.

Sample code

>>> from pyRserve import connect
>>> conn = connect('your R server')
>>> conn.r('1+1')                # direct evaluation of a statement
2
>>> conn.r.myList = [1, 2, 3]    # bind a list within R to variable 'myList'

>>> conn.r('func1 <- function(v) { v*2 }') # create a function in R
>>> conn.r.func1(4)                        # call the function in R
16

Changes in V 0.5.2

Fixed bug with 32bit integer conversion on 64bit machines. Upgrade highly recommended!

Documentation and Support

The documentation for pyRserve is available at http://packages.python.org/pyRserve
The corresponding Google news group can be found at http://groups.google.com/group/pyrserve

pyRserve 0.5.0 released

Ralph Heinkel — Wed, 12 Oct 2011 13:22:00 GMT

The latest version is now available via pypi (easy_install -U pyRserve).

About pyRserve

pyRServe is a library for connecting Python to an R process running under Rserve. Through such a connection variables can be get and set in R from Python, and also R-functions can be called remotely.

Changes in V 0.5.0

Renamed pyRserve.rconnect() to pyRserve.connect(). The former still works but shows a DeprecationWarning
String evaluation should now only be executed on the namespace directly, not on the connection object anymore. The latter still works but shows a DeprecationWarning.
New kw argument atomicArray=True added to pyRserve.connect() for preventing single valued arrays from being converted into atomic python data types.

Documentation and Support

The documentation for pyRserve is available at http://packages.python.org/pyRserve
The corresponding Google news group can be found at http://groups.google.com/group/pyrserve