Asynchronous communication
Important: Read Part I of this post before continuing.
In Part I, we looked at how to call Python functions from Elixir using Erlport!
ErlPort is an Elixir library which makes it easier to connect Erlang to a number of other programming languages using the Erlang port protocol.
However, as you may have noticed, the calling process had to wait for Python to finish processing and return the results. This works for operations that are instant and doesn’t take much time. Otherwise, it’s not much fun when you are running an intensive Python function. Because your Elixir program is gonna hang waiting for the Python code to finish.
For operations that take a while to complete we need a way to call the Python function and get notified in our elixir code when the function is complete — similar to how we write elixir program with processes talking to each asynchronously. This is useful, especially in cases when you want an Elixir process to control/supervise several Python operations. We can run the Python functions concurrently which makes life a little better.
In this post, we’ll explore how to achieve this — Asynchronous communication between Python and Elixir!
We can reason about the challenge the same way we do in communicating between Elixir processes — by using cast — which allows us to communicate between processes asynchronously.
Previously, we used erlport’s [:python.call/4](http://erlport.org/docs/python.html#python-call-4) mostly when calling functions from Python. This is synchronous and the elixir program had to wait for the python function to finish and send response.
In this post, we will use [:python.cast/2](http://erlport.org/docs/python.html#python-cast-2)
Understanding :python.cast/2
:python.cast/2 works the same as [GenServer.cast/2](https://hexdocs.pm/elixir/GenServer.html#cast/2). You give it process id (or registered process name) and the message to send to that process. Unlike :python.call variant, :python.cast is used to send message instead of calling a Python function. Same way your Elixir processes are sent messages. Erlport provides a mechanism for us to receive messages sent to the python instance.
Erlport Python Module
Erlport offers a python library that helps with working with elixir data types and processes. You can install erlport using pip.
pip install erlport #don't run this yet!
However, the Erlport Elixir repo already comes with the Python library which are automatically loaded at runtime. So you don’t have to install the Erlport Python library if you are using Erlport through Elixir.
Below is how data is mapped between Erlang/Elixir and Python
Erlang/Elixir data types mapped to Python data types (http://erlport.org/docs/python.html)
Python data types mapped to Erlang/Elixir data types (http://erlport.org/docs/python.html)
Handling Message Sent from Elixir to Python
For Elixir processes you have a receive block in which all messages are handled. Or the handle_call, handle_cast and handle_info functions if you are using GenServer. Erlport provides a similar mechanism for handling messages sent to the Python process. _set_message_handler_ is the function. It’s in the erlport.erlang module. It takes a Python function with one argument-message i.e the incoming message. Erlport will then call the function passed to set_message_handler whenever a message is received. It’s like a callback function.
Sending Message from Python to Elixir
Since an Elixir process isn’t aware of who sent a cast message, we have to find a way to know which process to send the result to once the python function completes. One way is to register the process id, of the Elixir process that you want the result sent to.
The Erlport python module provides [cast](http://erlport.org/docs/python.html#erlport-erlang-cast) function for sending a message to Elixir process. With this we can send asynchronously message back to the elixir process when the function is done!
Putting it all together
Open Terminal and create a new Elixir project using mix
$ mix new elixir_python
Add dependency
Add erlport to your dependencies mix.exs
defp deps do[{:erlport, "~> 0.9"},]end
Then install project dependencies.
$ cd elixir_python$ mix deps.get
Create priv/python directory where you’ll keep our python modules
$ mkdir -p priv/python
Create an elixir module wrapper for Erlport related functions.
#lib/python.exdefmodule ElixirPython.Python do
@doc """
Start python instance with custom modules dir priv/python
"""
**def** start() **do**
path = \[
:code.priv\_dir(:elixir\_python),
"python"
\]|> Path.join()
{:ok, pid} = :python.start(\[
{:python\_path, to\_charlist(path)}
\])
pid
**end**
**def** call(pid, m, f, a \\\\ \[\]) **do**
:python.call(pid, m, f, a)
**end**
**def** cast(pid, message) **do**
:python.cast(pid, message)
**end**
**def** stop(pid) **do**
:python.stop(pid)
**end**
end
In order for Elixir process to receive message asynchronously, we will create a simple GenServer module
#lib/python_server.exdefmodule ElixirPython.PythonServer douse GenServeralias ElixirPython.Python
def start_link() do GenServer.start_link(__MODULE__, [])end
def init(args) do #start the python session and keep pid in statepython_session = Python.start()#register this process as the message handlerPython.call(python_session, :test, :register_handler, [self()]){:ok, python_session}end
def cast_count(count) do {:ok, pid} = start_link()GenServer.cast(pid, {:count, count})**end
def** call_count(count) do {:ok, pid} = start_link()# :infinity timeout only for demo purposesGenServer.call(pid, {:count, count}, :infinity)**end
def** handle_call({:count, count}, from, session) do result = Python.call(session, :test, :long_counter, [count]){:reply, result, session}end
def handle_cast({:count, count}, session) do Python.cast(session, count){:noreply, session}end
def handle_info({:python, message}, session) do IO.puts("Received message from python: #{inspect message}")
#stop elixir process
{:stop, :normal, session}
end
def terminate(_reason, session) do Python.stop(session):okend
end
Now lets create the python module priv/python/test.py
#priv/python/test.pyimport timeimport sys#import erlport modules and functionsfrom erlport.erlang import set_message_handler, castfrom erlport.erlterms import Atom
message_handler = None #reference to the elixir process to send result to
def cast_message(pid, message):cast(pid, message)
def register_handler(pid):#save message handler pidglobal message_handlermessage_handler = pid
def handle_message(count):try:print "Received message from Elixir"print countresult = long_counter(count)if message_handler:#build a tuple to atom {:python, result}cast_message(message_handler, (Atom('python'), result))except Exception, e:# you can send error to elixir process here too# print epassdef long_counter(count=100):#simluate a time consuming python functioni = 0data = []while i < count:time.sleep(1) #sleep for 1 secdata.append(i+1)i = i + 1return data
set_message_handler(handle_message) #set handle_message to receive all messages sent to this python instance
Compile!
$ mix compile
Run!
$ iex -S mixiex(1)> ElixirPython.PythonServer.call_count(4)[1, 2, 3, 4] #printed after waiting 4seciex(2)> ElixirPython.PythonServer.call_count(10)[1, 2, 3, 4, 5, 6, 7, 8, 9, 10] #printed after waiting 10sec!
Also notice that call_count causes iex to hang till we get the back result from python
Now let’s call same python function asynchronously!
iex(3)> ElixirPython.PythonServer.cast_count(4):okReceived message from Elixir4Received message from python: [1, 2, 3, 4] # printed after 4sec, no waitiex(4)> ElixirPython.PythonServer.cast_count(10):okReceived message from Elixir10iex(5)>nilReceived message from python: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] #printed after 10sec without blocking
Notice how we can continue to work in iex and get the result once our python function completes. Try callingcast_count with different numbers — large and small.
That is it! Now you can communicate between Elixir and Python asynchronously.
Happy coding!