Python Run-Time Environment

Contents

• Installation
• Configure WASD
• Example Scripts
• Web Server Gateway Interface
• Leveraging PyRTE
• WATCHing Scripts
• Problems
• Releases
• Acknowlegements

PyRTE is an interface to a Python interpreter engine and environment for the WASD VMS Web server. It is designed to be able to be used in standard CGI and CGIplus/RTE persistent scripting modes. The persistent modes (default) provide a ~35x (yes, an approximate thirty-five times) improvement in script activation times (compared to CGI/command-line) and comparably reduced load on both server and system. Note that this package does not contain the Python interpreter or kit, that has to be obtained and installed separately (a simple matter as it is provided via a PCSI package).

PyRTE is linked against the Python interpreter shareable image and so shares a set of Python capabilities in common with command-line OpenVMS Python. Alpha (AXP) and Itanium object modules are provided for the VMS platforms available for Python.

PyRTE has been developed and tested on V8.4, Alpha and Itanium, using WASD v12.n and VMS Python 2.7 kits originally ported by Jean-François Piéronne, and now off-line.

FYI: Beginning May 2023, PyRTE underwent significant rework prompted by the release VSI Python v3.10 — Itanium only. The core PyRTE code now supports both Python 2.7 and 3.10 as independent builds and executables. PyRTE reports itself as v2.n.n for the Python 2.7 build, and as v3.n.n for Python 3.10.

Installation

• If Python for OpenVMS is not currently installed, or if the version is not up-to-date, obtain the latest kit from

  https://vmssoftware.com/products/python/

  and install according to the instructions available.

• When updating it is suggested that any existing PyRTE kit be renamed out of the way before installing this kit. After successful installation and testing the old kit may be deleted.

  $ RENAME WASD_ROOT:[SRC]PYTHON.DIR WASD_ROOT:[SRC]PYTHON_nnn.DIR
  

• Obtain the PyRTE kit from

  https://wasd.vsm.com.au/wasd/

• UNZIP the kit

  $ SET DEFAULT WASD_ROOT:[000000]
  $ UNZIP "-V" location:PYRTEnnn.ZIP
  

• Note that this performs a link-only build against the supplied object modules. Compilation is also available (just leave-off the LINK).

  $ SET DEFAULT WASD_ROOT:[SRC.PYTHON]
  $ @BUILD_PYRTE3 LINK
  

  For existing Python 2.7:

  $ @BUILD_PYRTE2 LINK
  

• Copy the PyRTE3 engine to the script executable location.

  $ COPY WASD_EXE:PYRTE3.EXE CGI_EXE:
  

  For existing Python 2.7:

  $ COPY WASD_EXE:PYRTE.EXE CGI_EXE:
  

• Configure the Web server.

• Start scripting  ☺
  Try the example scripts.

Although there are experimentation modes built into PyRTE basically it will be used as a WASD Run-Time Environment providing persistent Python interpreter(s) and environment(s) for Python scripts and applications. Benchmarking indicates the CGIplus/RTE use reduces activation time to 3% of CGI (yes, that's correct, by a factor 35 - a python interpreter is quite expensive to instantiate). There are subtle differences in the way CGIplus and RTE parse and provide the PATH_INFO data (see the "WASD Scripting Overview" for detail).

Note: Python 2.n built and used PYRTE.EXE whereas Python 3.n builds and uses PYRTE3.EXE (in line with python3 nomenclature).

Configuration and Mapping

• Server global configuration can be used to indicate which file types should activate the Python engine and how perform non-script access.

  # WASD_CONFIG_GLOBAL
  [DclScriptRunTime]
  .PY $CGI-BIN:[000000]PYRTE3
  [AddType]
  .PY   text/plain  Python source
  .PYC  application/octet-stream  Python byte-code
  .PYO  application/octet-stream  Python optimised byte-code
  

  For existing Python 2.7:

  # WASD_CONFIG_GLOBAL
  [DclScriptRunTime]
  .PY $CGI-BIN:[000000]PYRTE
  

  Note: [DclScriptRunTime] and mapping rules described below may interact in unexpected ways.

• The following rules require the Python script files to be located in the site administrator controlled /cgi-bin/ path.

  # WASD_CONFIG_MAP 
  exec /py-bin/* (cgi_exe:pyrte3)/cgi-bin/* \
       script=syntax=unix script=query=none map=once ods=5
  

  For existing Python 2.7:

  # WASD_CONFIG_MAP 
  exec /py-bin/* (cgi_exe:pyrte)/cgi-bin/* \
       script=syntax=unix script=query=none map=once ods=5
  

• Alternatively, to automatically map scripts ending in .py to the RTE engine.

  # WASD_CONFIG_MAP  for automatic RTE usage
  map /cgi-bin/*.py* /py-bin/*.py*
  exec /py-bin/* (cgi_exe:pyrte3)/cgi-bin/* \
         script=syntax=unix script=query=none map=once ods=5
  

  For existing Python 2.7:

  exec /py-bin/* (cgi_exe:pyrte)/cgi-bin/* \
         script=syntax=unix script=query=none map=once ods=5
  

• The following rule would allow .py type files anywhere in the mapped directory structure to be executed. This may be what is desired but might be dangerous in some circumstances.

  # WASD_CONFIG_MAP 
  exec /web/**.py* /web/*.py* \
       script=syntax=unix script=query=none map=once ods=5
  

• It is a general principal that Web applications requiring write access to parts of the file-system be run in a unique account with those file-system areas being owned by and having protections associated with that account. WASD provides this using its PERSONA facility (see the "WASD Scripting Overview" for detail).

  # WASD_CONFIG_MAP 
  exec /appx/**.py* /appx_root/*.py* \
       script=syntax=unix script=query=none map=once ods=5 \
       script=as=USERX
  

• To provide in-situ activation of the example scripts below use the following configuration.

  # WASD_CONFIG_MAP 
  exec /py-bin/* (cgi_exe:pyrte3)/wasd_root/src/python/scripts/* \
       script=syntax=unix script=query=none map=once ods=5
  

  For existing Python 2.7:

  exec /py-bin/* (cgi_exe:pyrte)/wasd_root/src/python/scripts/* \
       script=syntax=unix script=query=none map=once ods=5
  

  Note that the location of the above example scripts has changed to [.SCRIPTS] since the v1.0 release.

Remember to

$ HTTPD /DO=MAP

Supporting Multiple Python Versions

It is sometimes useful, sometimes necessary, to maintain multiple versions of Python on a system, and also support multiple versions to web applications (scripts). It is not difficult to have PyRTE activate a specific version by wrapping the executable in a procedure that sets up the appropriate Python environment in a process-local context.

$! CGI_BIN:PYRTE3.COM
$ define /proc TMPDIR "/wasd_root/scratch"
$ pyrte3 == "$cgi_exe:pyrte3.exe"
$ pyrte3

$! CGI_BIN:PYRTE2.COM
$! activate JFP Python 2.7
$ @disk$jfplib0020i:[000000]lib_logicals.com
$ @disk$jfppy1400i:[000000]python_logicals.com
$ @python_root:[vms]setup
$ pyrte == "$cgi_exe:pyrte.exe"
$ pyrte

Each of these is mapped for independent activation based on the request path.

# WASD_CONFIG_MAP
exec /py-bin/* (@cgi_bin:pyrte3.com)/wasd_root/src/python/scripts/* \
     script=syntax=unix map=once script=query=none ods=5
exec /py2-bin/* (@cgi_bin:pyrte2.com)/volume**/wasd_root/src/python/scripts/* \
     script=syntax=unix map=once script=query=none ods=5

This is one approach. A little creativity would yield others.

Example Scripts

After configuration (as described immediately above) the following scripts may be used to confirm the environment is functioning.

• /py-bin/pyrte_test1.py
• /py-bin/pyrte_test2.py
• /py-bin/pyrte_test3.py  (this tests an error handler and therefore displays an error!)
• /py-bin/pyrte_test4.py

Some of these examples also support CGIplus mode for comparison purposes.

Of course the Python code in the script may be inspected from the source directory:

/wasd_root/src/python/scripts/

PyRTE contains a Web Server Gateway Interface v1.0 (WSGI). This is a Python standard specification for web servers and application servers to communicate with web applications (though it can also be used for more than that).

• http://www.wsgi.org/wsgi/
• http://www.python.org/dev/peps/pep-0333/

The WSGI code has been developed under the sponsorship of SysGroup

http://www.sysgroup.fr/

The following is an example of a simple (and classic) WSGI application being activated using the PyRTE wasd.wsgi_run() function.

def hello_world (environ, start_response):
    status = '200 OK'
    response_headers = [('Content-type','text/plain')]   
    start_response(status, response_headers)
    return ['Hello world!\n']

import wasd
wasd.wsgi_run(hello_world)

WSGI Output Buffering

PEP 333 specifies that WSGI should not buffer output and PyRTE complies but some tools (like Mercurial) don't buffer internally themselves and end up sending many very small records. This is obviously inefficient. If buffered the throughput boost for Mercurial is about x4. Well worth having for specific Web applications.

This forced buffering may be enabled on a per-application basis using a mapping rule, for example

set /mercurial/* script=param=PYRTE=/WSGI=BUFFER

$ DEFINE /JOB PYRTE_WSGI_FORCE_BUFFERING TRUE

Example WSGI Scripts

The following scripts may be used to confirm the environment is functioning.

• /py-bin/wsgi_test1.py
• /py-bin/wsgi_test2.py
• /py-bin/wsgi_test3.py
• /py-bin/wsgi_test4.py

The Python code in the script may be inspected from the source directory:

/wasd_root/src/python/scripts/

There are also equivalent wsgiref (the Python reference implementation) scripts available for comparison. Just add ref to the wsgi in the script name.

Leveraging PyRTE

Python scripts activated using the default persistent engine generally have low-latency, low-system-impact characteristics and seem to perform reliably and efficiently. The PyRTE generally handles all the WASD server interaction requirements. However there may be occasions where the application itself benefits from being persistent (many Python Web application are written with this in mind) or where more control needs to be exercised by the application designer. The WASD PyRTE provides a WASD Python module that provides an API for some of these aspects. Rather than clutter this document with such arcane detail the description may be found in the prologue to the PyRTE source code:

/wasd_root/src/python/pyrte.c

An example of where an application benefits from an explicitly CGIplus activation is MoinMoin:

• https://moinmo.in
• https://www.vmspython.org/DownloadAndInstallationMoinMoin

MoinMoin is generally run independently to a system's primary Web server using the embedded Web server available for Python. Reverse proxy is often then use to provide and control access through to the persistent MoinMoin server.

Using the same principle of keeping MoinMoin persistent (for the obvious latency and efficiency benefits) it can be run and controlled directly by the WASD Web server using a CGIplus wrapper DCL procedure and a little Python glue (directly derived from persistence code already present in the MoinMoin package).

/wasd_root/src/python/moincgiplus.com

The same principles may be applied to any application environment (third-party or in-house) with similar characteristics. Python applications running as CGIplus scripts using the PyRTE must be activated via DCL wrappers as illustrated above (in part due to mapping requirements).

With CGIplus persistent applications configuration or code changes will in all probability require a 'restart' of the application using one or other of:

$ HTTPD/DO=PURGE
$ HTTPD/DO=DELETE

Debugging is always fun  ☹

PyRTE provides some execution data via the WATCH report [X]Script item. For example (note the SCRIPTs):

|13:37:44.55 SERVICE  1749 009001 CONNECT    VIRTUAL wasd.vsm.com.au:443|
|13:37:44.55 REQUEST  4418 009001 REQUEST    GET /py-bin/pyrte_test2.py|
|13:37:44.55 CACHE    0600 009001 RESPONSE   CACHE search path 23E5BF0E9DBCDFE3B01647C116E79A09|
|13:37:44.55 DCL      1611 009001 RESPONSE   SCRIPT as HTTP$NOBODY RTE /py-bin/pyrte_test2.py wasd_root:[src.python.scripts]pyrte_test2.py ($cgi_exe:pyrte3.exe)|
|13:37:45.24 DCL      8396 009001 SCRIPT     PYRTE IA64-3.0.0 Python 3.10.0 (default, Nov 25 2021, 10:52:09) [C]|
|13:37:45.25 DCL      8396 009001 SCRIPT     RTE caching /py-bin/pyrte_test2.py /WASD_ROOT/src/python/SCRIPTS/pyrte_test2.py|
|13:37:45.25 DCL      8396 009001 SCRIPT     CACHE new 1/1|
|13:37:45.25 DCL      8396 009001 SCRIPT     LOAD /py-bin/pyrte_test2.py /WASD_ROOT/src/python/SCRIPTS/pyrte_test2.py|
|13:37:45.25 DCL      8396 009001 SCRIPT     CODE /WASD_ROOT/src/python/SCRIPTS/pyrte_test2.py|
|13:37:45.26 DCL      8396 009001 SCRIPT     EVAL /py-bin/pyrte_test2.py|
|13:37:47.75 DCL      8396 009001 SCRIPT     PYRTE IA64-3.0.0 USAGE:1/1 REAL:00:00:02.51 CPU:2.49 DIO:473 BIO:2437 FAULTS:957 PGFL:1966832/2%|
|13:37:47.75 GZIP     0608 009001 RESPONSE   DEFLATE 3366->1521 bytes, 45% (261kB)|
|13:37:47.75 NETIO    0514 009001 RESPONSE   NETWORK %X0000002C (%SYSTEM-F-ABORT, abort) (non-blocking)|
|13:37:47.75 REQUEST  1435 009001 REQUEST    STATUS 200 (OK) rx:344 tx:4911 bytes 4.205s 1,249 B/s|

Any reportable script error is indicated.

|13:43:44.30 DCL      8396 011001 SCRIPT     CODE /WASD_ROOT/src/python/SCRIPTS/pyrte_test3.py|
|13:43:44.31 DCL      8396 011001 SCRIPT     EVAL /py-bin/pyrte_test3.py|
|13:43:44.31 DCL      8396 011001 SCRIPT     ERROR <class 'RuntimeError'> 'logical name PYRTE_CACHE_ENTRY not defined'|
|13:43:44.32 DCL      8396 011001 SCRIPT     PYRTE IA64-3.0.0 USAGE:2/2 REAL:00:00:00.01 CPU:0.02 DIO:1 BIO:61 FAULTS:2 PGFL:1961552/2%|

It is also possible to add items from within the script Python code using the wasd.WATCH() function. The example /py-bin/pyrte_test1.py script contains the statement  wasd.WATCH('and hello [x]Script')

|13:48:12.43 DCL      8396 013001 SCRIPT     RTE caching /py-bin/pyrte_test1.py /WASD_ROOT/src/python/SCRIPTS/pyrte_test1.py|
|13:48:12.43 DCL      8396 013001 SCRIPT     CACHE unused 3/3|
|13:48:12.43 DCL      8396 013001 SCRIPT     LOAD /py-bin/pyrte_test1.py /WASD_ROOT/src/python/SCRIPTS/pyrte_test1.py|
|13:48:12.44 DCL      8396 013001 SCRIPT     CODE /WASD_ROOT/src/python/SCRIPTS/pyrte_test1.py|
|13:48:12.44 DCL      8396 013001 SCRIPT     EVAL /py-bin/pyrte_test1.py|
|13:48:12.44 DCL      8396 013001 SCRIPT     WATCH and hello [x]Script|
|13:48:12.44 DCL      8396 013001 SCRIPT     PYRTE IA64-3.0.0 USAGE:2/4 REAL:00:00:00.01 CPU:0.00 DIO:3 BIO:37 FAULTS:2 PGFL:1961088/2%|

• With the PyRTE kit ... Mark.Daniel@wasd.vsm.com.au
• With the VSI Python kit ... VSI Python Forum
• The info-WASD mailing list

Unfortunately the author of the PyRTE interface is such a Python novice he is not in any position to answer queries about Python "programming" or usage. Suggest asking VMS-specific-Python questions at the

VSI Python Forum on the general VSI OpenVMS Forum

If there's an obvious behavioural problem with PyRTE then contact the author or use the info-WASD mailing list.

Releases

v3.1.0  18-APR-2024

•  VSI X86VMS PYTHON A3.10-0RELEASE008

• One script tweaked for Python 3

v3.0.0  20-MAY-2023

•  VSI I64VMS PYTHON A3.10-0RELEASE001

•  in line with other WASD apps, moved under Apache License

•  this is a dot-zero release – YMMV

v2.0.0  20-MAY-2023

•  Python version 2.7.18

v1.1.12  30-JUL-2017

•  wasd.cgi_callout()

•  bugfix

⋮

v1.0.0  22-APR-2007

•  initial

Many thanks to Jean-François Piéronne for his initial port of Python to VMS.
VSI also have provided an official port of v3.10 for Itanium and now x86-64.