Edit detail for SandBox Sockets in SPAD revision 1 of 2

changed:
-
SocketsInSpad

This is an attempt to add GCL-based socket support to SPAD
and to use it to implement a very simple web server.

First we create a Lisp interface for the GCL socket server-mode.
Unfortunately there is no common socket interface for Lisp.

GCL Sockets

  The documentation for GCL sockets is available here::

    $ info gcl-si.info

in the 'gcl/info' directory of the GCL source distribution.
Look under the heading 'GCL specific':

Function: SOCKET (port &key host server async myaddr myport daemon) --
  Establishes a socket connection to the specified PORT under a
  variety of circumstances.

  If HOST is specified, then it is a string designating the IP
  address of the server to which we are the client.  ASYNC specifies
  that the connection should be made asynchronously, and the call
  return immediately.  MYADDR and MYPORT can specify the IP address
  and port respectively of a client connection, for example when the
  running machine has several network interfaces.

  If SERVER is specified, then it is a function which will handle
  incoming connections to this PORT.  DAEMON specifies that the
  running process should be forked to handle incoming connections in
  the background.  If DAEMON is set to the keyword PERSISTENT, then
  the backgrounded process will survive when the parent process
  exits, and the SOCKET call returns NIL.  Any other non-NIL setting
  of DAEMON causes the socket call to return the process id of the
  backgrounded process.  DAEMON currently only works on BSD and
  Linux based systems.

  If DAEMON is not set or nil, or if the socket is not a SERVER
  socket, then the SOCKET call returns a two way stream.  In this
  case, the running process is responsible for all I/O operations on
  the stream.  Specifically, if a SERVER socket is created as a
  non-DAEMON, then the running process must LISTEN for connections,
  ACCEPT them when present, and call the SERVER function on the
  stream returned by ACCEPT.

Axiom Sockets

  Currently the socket support in Axiom is implemented as an
interface
"src/interp/sockio.lisp.pamphlet":http://wiki.axiom-developer.org/axiom--test--1/src/interp/SockioLisp
to Axiom's external "C" socket code:
"src/lib/sockio-c.c":http://wiki.axiom-developer.org/axiom--test--1/src/lib/SocketCC
But this is currently only available in Linux version of Axiom.

The interface should really make use of the socket support built-in
to the underlying Lisp.

A Simple Lisp Web Server

  For our current purposes we can define a simple interface based
on some code originally provided by Camm Maguire's
"simple http server in lisp":http://lists.nongnu.org/archive/html/axiom-developer/2005-04/msg00141.html

\begin{lisp}
;; file: http-test.lisp
(defun bar (p fn) 
  (let ((s (si::socket p :server fn))) 
        (tagbody l 
                (when (si::listen s) 
                        (let ((w (si::accept s))) 
                                (foo w))) 
                (sleep 3) 
                (go l))))

(defun foo (s) 
  (let* ((get (read s nil 'eof)) 
         (fn (and (eq get 'get) (string-downcase (read s nil 'eof))))
         )
    (format t "Got ~S~%~%" fn)
    (let ((fn (when (probe-file fn) fn) )))
    (format s "HTTP/1.1 ~S~%~%" (if fn 200 403))
    (format t "HTTP/1.1 ~S~%~%" (if fn 200 403))
    (when fn
      (if (pathname-name (pathname fn))
          (with-open-file (q fn) (si::copy-stream q s))
        (dolist (l (directory fn)) (format s "~a~%" (namestring l)))))
    (close s)))
\end{lisp}

To start the HTTP server on port 8085 we call::

  >(bar 8085 #'foo)

or in Axiom we can use the commands::

  )lisp (load "http-test.lisp")
  )lisp (bar 8085 #'foo)

then access with the url::

  http://localhost:8085/directory/or/file

Sockets in SPAD

  Our objective is to be able to write a more sophisticated
web server in SPAD. To do this we still need some Lisp coding
as an interface to the GCL socket functions. Perhaps this is
possible using the '$Lisp' package call like this::

  1) -> SI_:_:SOCKET(p ...)$Lisp

but I am unable to find a way to specify a keyword like ':server'.

So here is a simple Lisp interface routine that does the job:
\begin{lisp}
(defun |SiSocket| (p spadfn)
  (si::socket p :server
    (function
      (lambda (w)
        (SPADCALL w spadfn)))))
\end{lisp}

SPADCALL

  'SPADCALL' calls the SPAD function 'spadfn' with the parameter 'w'.

There is an example of similar use of 'SPADCALL' here:
"spad.lisp.pamphlet":http://wiki.axiom-developer.org/axiom--test--1/src/interp/SpadLisp

\begin{lisp}
(defun |MySort| (seq spadfn)
    (sort (copy-seq seq) (function (lambda (x y) (SPADCALL X Y SPADFN)))))
\end{lisp}

which can be called in Axiom for example like this:
\begin{axiom}
L:SEX:=[3::SEX,1::SEX,(-4)::SEX,2::SEX]
MySort(L,<$Integer)$Lisp
\end{axiom}

More GCL Socket Functions

  We also need these extra functions

\begin{lisp}
(defun |SiListen| (s) (si::listen s))

(defun |SiAccept| (s) (si::accept s))

(defun |SiCopyStream| (q s) (si::copy-stream q s))
\end{lisp}

Function: ACCEPT (stream) --
     Creates a new two-way stream to handle an individual incoming
     connection to STREAM, which must have been created with the SOCKET
     function with the SERVER keyword set.  ACCEPT should only be
     invoked when LISTEN on STREAM returns T.  If the STREAM was
     created with the DAEMON keyword set in the call to SOCKET, ACCEPT
     is unnecessary and will be called automatically as needed.

Function: LISTEN (&optional (stream *standard-input*)) --
     Package:LISP

     Returns T if a character is available on STREAM; NIL otherwise.
     This function does not correctly work in some versions of GCL
     because of the lack of such mechanism in the underlying operating
     system.

Function: COPY-STREAM (in-stream out-stream) --
     Package:SI

     GCL specific: Copies IN-STREAM to OUT-STREAM until the end-of-file
     on IN- STREAM.

Server-side Sockets

  Now we define a package in SPAD that sets up the socket server.

\begin{spad}
)abbrev package SISOCK SiSocket
SiSocket: with
    socketServer: (Integer,SExpression->Void) -> Void
  == add
    -- exported --
    socketServer(port:Integer,server:SExpression->Void):Void ==
      s:=SiSocket(port,server)$Lisp
      while not null?(SiListen(s)$Lisp)$SExpression repeat
        w := SiAccept(s)$Lisp
        server(w)
        SLEEP(3)$Lisp
\end{spad}

Web Server in SPAD

  Then we can define an simple HTTP (web) server as follows:
\begin{spad}
)abbrev package LHTTPD LittleHttpDeamon
LittleHttpDeamon: with
    server: SExpression -> Void
  == add
    server(s:SExpression):Void ==
      get := READ(s,NIL$Lisp,'eof)$Lisp
      if not null? EQ(get,'get)$Lisp then
        fn := READ(s,NIL$Lisp,'eof)$Lisp
        fn := PROBE_-FILE(fn)$Lisp
        if not null? fn then
          -- header: ok
          WRITE_-LINE("HTTP/1.1 200",s)$Lisp
          WRITE_-LINE("",s)$Lisp
          if not null? PATHNAME_-NAME(PATHNAME(fn)$Lisp)$Lisp then
            -- display contents of file
            q:=OPEN(fn)$Lisp
            SiCopyStream(q,s)$Lisp
            CLOSE(q)$Lisp
          else
            -- directory listing
            for l in destruct DIRECTORY(fn)$Lisp repeat
              WRITE_-LINE(string(NAMESTRING(l)$Lisp),s)$Lisp
        else
          -- header: not found 
          WRITE_-LINE("HTTP/1.1 403",s)$Lisp
          WRITE_-LINE("",s)$Lisp
      CLOSE(s)$Lisp
\end{spad}

And then start the server like this:
\begin{axiom}
socketServer(8085,server$LHTTPD)
\end{axiom}

If this worked, a new web server on port 8085 would be launched
on the axiom-developer.org server. That would cause the 'Save'
or 'refresh' of this page to appear to hang (until the AXIOMsys
process is aborted by the system time-out) and this page would
then appear to be broken.

Unfortunately (or fortunately for readers of this page) we still
have some debugging to do.

SocketsInSpad?

This is an attempt to add GCL-based socket support to SPAD and to use it to implement a very simple web server.

First we create a Lisp interface for the GCL socket server-mode. Unfortunately there is no common socket interface for Lisp.

GCL Sockets

The documentation for GCL sockets is available here:

    $ info gcl-si.info

in the gcl/info directory of the GCL source distribution. Look under the heading 'GCL specific':

Function: SOCKET (port &key host server async myaddr myport daemon)

Establishes a socket connection to the specified PORT under a variety of circumstances.

If HOST is specified, then it is a string designating the IP address of the server to which we are the client. ASYNC specifies that the connection should be made asynchronously, and the call return immediately. MYADDR and MYPORT can specify the IP address and port respectively of a client connection, for example when the running machine has several network interfaces.

If SERVER is specified, then it is a function which will handle incoming connections to this PORT. DAEMON specifies that the running process should be forked to handle incoming connections in the background. If DAEMON is set to the keyword PERSISTENT, then the backgrounded process will survive when the parent process exits, and the SOCKET call returns NIL. Any other non-NIL setting of DAEMON causes the socket call to return the process id of the backgrounded process. DAEMON currently only works on BSD and Linux based systems.

If DAEMON is not set or nil, or if the socket is not a SERVER socket, then the SOCKET call returns a two way stream. In this case, the running process is responsible for all I/O operations on the stream. Specifically, if a SERVER socket is created as a non-DAEMON, then the running process must LISTEN for connections, ACCEPT them when present, and call the SERVER function on the stream returned by ACCEPT.

Axiom Sockets

Currently the socket support in Axiom is implemented as an interface src/interp/sockio.lisp.pamphlet to Axiom's external "C" socket code: src/lib/sockio-c.c But this is currently only available in Linux version of Axiom.

The interface should really make use of the socket support built-in to the underlying Lisp.

A Simple Lisp Web Server

For our current purposes we can define a simple interface based on some code originally provided by Camm Maguire's simple http server in lisp

lisp

;; file: http-test.lisp
(defun bar (p fn) 
  (let ((s (si::socket p :server fn))) 
        (tagbody l 
                (when (si::listen s) 
                        (let ((w (si::accept s))) 
                                (foo w))) 
                (sleep 3) 
                (go l))))

(defun foo (s) 
  (let* ((get (read s nil 'eof)) 
         (fn (and (eq get 'get) (string-downcase (read s nil 'eof))))
         )
    (format t "Got ~S~%~%" fn)
    (let ((fn (when (probe-file fn) fn) )))
    (format s "HTTP/1.1 ~S~%~%" (if fn 200 403))
    (format t "HTTP/1.1 ~S~%~%" (if fn 200 403))
    (when fn
      (if (pathname-name (pathname fn))
          (with-open-file (q fn) (si::copy-stream q s))
        (dolist (l (directory fn)) (format s "~a~%" (namestring l)))))
    (close s)))

lisp

; compiling file "/var/aw/var/LatexWiki/304369446719046572-25px001.lisp" (written 14 JUL 2013 10:50:21 AM):
; compilation aborted after 0:00:00.003

   >> System error:
   The value NIL
is not of type
  (OR (VECTOR CHARACTER) (VECTOR NIL) BASE-STRING PATHNAME STREAM).

To start the HTTP server on port 8085 we call:

  >(bar 8085 #'foo)

or in Axiom we can use the commands:

  )lisp (load "http-test.lisp")
  )lisp (bar 8085 #'foo)

then access with the url:

  http://localhost:8085/directory/or/file

Sockets in SPAD

Our objective is to be able to write a more sophisticated web server in SPAD. To do this we still need some Lisp coding as an interface to the GCL socket functions. Perhaps this is possible using the $Lisp package call like this::

1) -> SI_:_:SOCKET(p ...)$Lisp

but I am unable to find a way to specify a keyword like :server.

So here is a simple Lisp interface routine that does the job:

lisp

(defun |SiSocket| (p spadfn)
  (si::socket p :server
    (function
      (lambda (w)
        (SPADCALL w spadfn)))))

lisp

; compiling file "/var/aw/var/LatexWiki/7125197138323642553-25px002.lisp" (written 14 JUL 2013 10:50:21 AM):
; compilation aborted after 0:00:00.001

   >> System error:
   The value NIL
is not of type
  (OR (VECTOR CHARACTER) (VECTOR NIL) BASE-STRING PATHNAME STREAM).

SPADCALL

SPADCALL calls the SPAD function spadfn with the parameter w.

There is an example of similar use of SPADCALL here: spad.lisp.pamphlet

lisp

(defun |MySort| (seq spadfn)
    (sort (copy-seq seq) (function (lambda (x y) (SPADCALL X Y SPADFN)))))

lisp

; compiling file "/var/aw/var/LatexWiki/5063368822247861847-25px003.lisp" (written 14 JUL 2013 10:50:21 AM):

; /var/lib/zope2.10/instance/axiom-wiki/var/LatexWiki/5063368822247861847-25px003.fasl written
; compilation finished in 0:00:00.004
Value = T

which can be called in Axiom for example like this:

fricas

L:SEX:=[3::SEX,1::SEX,(-4)::SEX,2::SEX]

(1)

Type: SExpression?

fricas

MySort(L,<$Integer)$Lisp

(2)

Type: SExpression?

More GCL Socket Functions

We also need these extra functions

lisp

(defun |SiListen| (s) (si::listen s))

(defun |SiAccept| (s) (si::accept s))

(defun |SiCopyStream| (q s) (si::copy-stream q s))

lisp

; compiling file "/var/aw/var/LatexWiki/1107245396490344730-25px005.lisp" (written 14 JUL 2013 10:50:21 AM):
; compilation aborted after 0:00:00.001

   >> System error:
   The value NIL
is not of type
  (OR (VECTOR CHARACTER) (VECTOR NIL) BASE-STRING PATHNAME STREAM).

Function: ACCEPT (stream)

Creates a new two-way stream to handle an individual incoming connection to STREAM, which must have been created with the SOCKET function with the SERVER keyword set. ACCEPT should only be invoked when LISTEN on STREAM returns T. If the STREAM was created with the DAEMON keyword set in the call to SOCKET, ACCEPT is unnecessary and will be called automatically as needed.

Function: LISTEN (&optional (stream standard-input))

Package:LISP

Returns T if a character is available on STREAM; NIL otherwise. This function does not correctly work in some versions of GCL because of the lack of such mechanism in the underlying operating system.

Function: COPY-STREAM (in-stream out-stream)

Package:SI

GCL specific: Copies IN-STREAM to OUT-STREAM until the end-of-file on IN- STREAM.

Server-side Sockets

Now we define a package in SPAD that sets up the socket server.

spad

)abbrev package SISOCK SiSocket
SiSocket: with
    socketServer: (Integer,SExpression->Void) -> Void
  == add
    -- exported --
    socketServer(port:Integer,server:SExpression->Void):Void ==
      s:=SiSocket(port,server)$Lisp
      while not null?(SiListen(s)$Lisp)$SExpression repeat
        w := SiAccept(s)$Lisp
        server(w)
        SLEEP(3)$Lisp

spad

   Compiling FriCAS source code from file 
      /var/lib/zope2.10/instance/axiom-wiki/var/LatexWiki/4690300564260941383-25px006.spad
      using old system compiler.
   SISOCK abbreviates package SiSocket 
------------------------------------------------------------------------
   initializing NRLIB SISOCK for SiSocket 
   compiling into NRLIB SISOCK 
   compiling exported socketServer : (Integer,SExpression -> Void) -> Void
Time: 0 SEC.

(time taken in buildFunctor:  0)

;;;     ***       |SiSocket| REDEFINED

;;;     ***       |SiSocket| REDEFINED
Time: 0 SEC.

   Cumulative Statistics for Constructor SiSocket
      Time: 0 seconds

   finalizing NRLIB SISOCK 
   Processing SiSocket for Browser database:
--->-->SiSocket(constructor): Not documented!!!!
--->-->SiSocket((socketServer ((Void) (Integer) (Mapping (Void) (SExpression))))): Not documented!!!!
--->-->SiSocket(): Missing Description
; compiling file "/var/aw/var/LatexWiki/SISOCK.NRLIB/SISOCK.lsp" (written 14 JUL 2013 10:50:21 AM):

; /var/aw/var/LatexWiki/SISOCK.NRLIB/SISOCK.fasl written
; compilation finished in 0:00:00.011
------------------------------------------------------------------------
   SiSocket is now explicitly exposed in frame initial 
   SiSocket will be automatically loaded when needed from 
      /var/aw/var/LatexWiki/SISOCK.NRLIB/SISOCK

Web Server in SPAD

Then we can define an simple HTTP (web) server as follows:

spad

)abbrev package LHTTPD LittleHttpDeamon
LittleHttpDeamon: with
    server: SExpression -> Void
  == add
    server(s:SExpression):Void ==
      get := READ(s,NIL$Lisp,'eof)$Lisp
      if not null? EQ(get,'get)$Lisp then
        fn := READ(s,NIL$Lisp,'eof)$Lisp
        fn := PROBE_-FILE(fn)$Lisp
        if not null? fn then
          -- header: ok
          WRITE_-LINE("HTTP/1.1 200",s)$Lisp
          WRITE_-LINE("",s)$Lisp
          if not null? PATHNAME_-NAME(PATHNAME(fn)$Lisp)$Lisp then
            -- display contents of file
            q:=OPEN(fn)$Lisp
            SiCopyStream(q,s)$Lisp
            CLOSE(q)$Lisp
          else
            -- directory listing
            for l in destruct DIRECTORY(fn)$Lisp repeat
              WRITE_-LINE(string(NAMESTRING(l)$Lisp),s)$Lisp
        else
          -- header: not found 
          WRITE_-LINE("HTTP/1.1 403",s)$Lisp
          WRITE_-LINE("",s)$Lisp
      CLOSE(s)$Lisp

spad

   Compiling FriCAS source code from file 
      /var/lib/zope2.10/instance/axiom-wiki/var/LatexWiki/6898618925161354916-25px007.spad
      using old system compiler.
   LHTTPD abbreviates package LittleHttpDeamon 
------------------------------------------------------------------------
   initializing NRLIB LHTTPD for LittleHttpDeamon 
   compiling into NRLIB LHTTPD 
   compiling exported server : SExpression -> Void
Time: 0 SEC.

(time taken in buildFunctor:  0)

;;;     ***       |LittleHttpDeamon| REDEFINED

;;;     ***       |LittleHttpDeamon| REDEFINED
Time: 0 SEC.

   Cumulative Statistics for Constructor LittleHttpDeamon
      Time: 0 seconds

   finalizing NRLIB LHTTPD 
   Processing LittleHttpDeamon for Browser database:
--->-->LittleHttpDeamon(constructor): Not documented!!!!
--->-->LittleHttpDeamon((server ((Void) (SExpression)))): Not documented!!!!
--->-->LittleHttpDeamon(): Missing Description
; compiling file "/var/aw/var/LatexWiki/LHTTPD.NRLIB/LHTTPD.lsp" (written 14 JUL 2013 10:50:21 AM):

; /var/aw/var/LatexWiki/LHTTPD.NRLIB/LHTTPD.fasl written
; compilation finished in 0:00:00.016
------------------------------------------------------------------------
   LittleHttpDeamon is now explicitly exposed in frame initial 
   LittleHttpDeamon will be automatically loaded when needed from 
      /var/aw/var/LatexWiki/LHTTPD.NRLIB/LHTTPD

And then start the server like this:

fricas

socketServer(8085,server$LHTTPD)

   >> System error:
   invalid number of arguments: 2

If this worked, a new web server on port 8085 would be launched on the axiom-developer.org server. That would cause the Save or refresh of this page to appear to hang (until the AXIOMsys? process is aborted by the system time-out) and this page would then appear to be broken.

Unfortunately (or fortunately for readers of this page) we still have some debugging to do.