How to make AWS Lambda and PostgreSQL

recently, in the AWS Lambda appeared native support for Python 2.7. For those who don't know what is AWS Lambda, refer to article on habré. The main advantage of using service — build a scalable serverless architecture. In this article I will tell how to make friends with AWS Lambda and PostgreSQL.

The main problem of using AWS Lambda, in conjunction with code in Python arises from the fact that the environment executing the code represents a short-lived instance (virtual server), in which it is impossible to add the necessary binary components. Moreover, the use of python code requires the creation of a so-called deployment package. A typical scheme of creating such a package is briefly as follows — to the directory used by the code, copy all the contents of the $VIRTUAL_ENV/lib/python2.7/site-packages, we pack in a zip file uploaded through the AWS console or command-line utilities in service, custom calling, etc. Interested refer to the documentation of the web service, since I intend to write is not about details, but about the specific problem.

If You try to use a similar scheme of work with sqlalchemy and PostgreSQL, when trying to call code that uses the query to the database, will receive approximately the following:

the

START RequestId: 23af180b-7b22-11e5-beac-0b062a6930ba Version: $LATEST
Unable to import module 'lambda': libpq.so.5: cannot open shared object file: No such file or directory

END RequestId: 23af180b-7b22-11e5-beac-0b062a6930ba
REPORT RequestId: 23af180b-7b22-11e5-beac-0b062a6930ba Duration: 0.32 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 38 MB 

This is due to the fact that psycopg2 — library for working with PostgreSQL in Python — is an extension that uses C code and shared library libpq.so part of the PostgreSQL distribution.

The decision in this case is (all steps were carried out on the Fedora 22):
1) Download the PostgreSQL source code from the page http://www.postgresql.org/ftp/source

2) Execute the command
the

tar xzvf postgresql-9.4.5.tar.gz
cd postgresql-9.4.5
./configure
make

3) Create and activate virtualenv, which we use to create the deployment package:
the

virtualenv ./env
source ./env/bin/activate

4) Continue working with the psycopg2 repository:
the

git clone -b maint_2_6 https://github.com/psycopg/psycopg2

5) From the subdirectory src/interfaces/libpq libpq copy the file.a directory with a cloned distro psycopg2. Run the build:
the

cd psycopg2
python setup.py build_ext -L ./

The alternative is to put libpq.a in directory /usr/lib64 or other defined environment variable LD_LIBRARY_PATH and the file setup.cfg to uncomment the parameter static_libpq=1.
4) Put the package:
the

python setup.py install

Now when you use this virtualenv to create a deployment package on a lack of shared library swearing will not.

Unfortunately, this approach cannot be called universal. In particular, I never managed to solve the problem with the pillow.Cm. UPDATE 2

I would be grateful if someone from Gabriela Express constructive ideas about it.

UPDATE. Here's an alternative solution:
1) When you build the package you specify the option as rpath /var/task/lib:

the

 python setup.py build_ext -R /var/task/lib

The deployment package, create a subdirectory of lib where we put a binary shared library:
the

(env)[random1st@localhost lambda]$ ls lib/
libpq.so libpq.so.5 libpq.so.5.7

UPDATE 2. Pip with some version supports specifying build options in requirements.txt. So you could:
the

(env)[random1st@localhost lambda]$ cat requirements.txt |grep option
Pillow==2.5.1 --global-option="build_ext" --global-option="--rpath=/var/task/lib"
psycopg2==2.5.3 --global-option="build_ext" --global-option="--rpath=/var/task/lib"

UPDATE 3. In the Amazon cloud connect takes place almost instantly, and in the course of this year promise a chance to tie the lambda to your VPC.
UPDATE 4. VPC tightened, but left the ability to connect to resources in your VPC. Amazon be Amazon.

Article based on information from habrahabr.ru