mmjobs



Thanks to this module it is possible to load several models in memory and execute them concurrently. In addition, other instances of Mosel might be started (either locally to the running system or remotely on another machine through the network) and used to run additional models controlled by the model that has started them. This means that the computing capacity of the running model is not restricted to the executing process. A general synchronization mechanism based on event queues as well as two specialized IO drivers are also provided in order to ease the implementation of parallel algorithms in Mosel.
To use this module, the following line must be included in the header of the Mosel model file:

 uses 'mmjobs'

Example

The following example shows how to compile, load, and then run a model from another model. After having started the execution, it waits for 60 seconds before stopping the secondary model if the latter has not yet finished.

model "mmjobs example"
uses "mmjobs","mmsystem"

declarations
 mymod: Model
 event: Event
end-declarations

                              ! Compile 'mymod.mos' to memory
 if compile("","mymod.mos","shmem:bim")<>0
 then
  exit(1)
 end-if

 load(mymod,"shmem:bim")      ! Load bim file from memory...
 fdelete("shmem:bim")         ! ... and release the memory block

                              ! Disable model output
 setdefstream(mymod,"","null:","null:")
 run(mymod)                   ! Start execution and
 wait(60)                     ! wait 1 min for an event

 if isqueueempty then         ! No event has been sent...
  writeln("Model too long: stopping it!")
  stop(mymod)                 ! ... stop the model then wait
  wait
 end-if
                              ! An event is available: model finished
 event:=getnextevent
 writeln("Exit status: ", getvalue(event))
 writeln("Exit code  : ", getexitcode(mymod))

 unload(mymod)
end-model

Control parameters

The following parameters are defined by mmjobs:

conntmpl
Default connection template.
fsrvdelay
Maximum wait time for findxsrvs.
fsrvnbiter
Number of iterations performed by findxsrvs.
fsrvport
UDP port used by findxsrvs.
jobid
ID of the current model.
keepalive
Keepalive timer setting.
nodenumber
ID of the current instance.
parentnumber
ID of the parent of the current instance.
sshcmd
SSH command for xssh driver.

Procedures and functions

Mosel instance management

The type Mosel is used to reference a Mosel instance. Before an instance can execute commands (like loading or running a model), it must be connected. Connecting an instance consists in starting an additional operating system process running Mosel: this is done by the connect function. To improve readability of the model source, one can use host aliases (defined by means of the sethostalias routine) to designate connection targets. Once work with a particular instance has been finished, the instance can be disconnected (disconnect): this terminates the process running Mosel (and releases all associated resources).

clearaliases
Delete all defined aliases.
connect
Connect a Mosel instance.
disconnect
Disconnect a Mosel instance.
findxsrvs
Search xprmsrv servers on the local network.
getaliases
Retrieve the list of all defined aliases.
getbanner
Get the banner displayed by an instance on startup.
gethostalias
Get the value of a host alias.
sethostalias
Define a host alias.

Model management

The type Model is used to reference a Mosel model. This section describes the procedures and functions available for model management: compilation of source model files, loading of bim files, execution and retrieval of model information. Note that before it can be used, a model has to be initialized by loading a bim file (load).

compile
Compile a source model.
getexitcode
Get the exit code of a model.
getid
Get the ID of a model or Mosel instance.
getstatus
Get the status of a model.
load
Load a Binary Model file.
reset
Reset a model.
run
Run a model.
setdefstream
Set default input/output streams of a model.
stop
Stop a running model.
unload
Unload a model.

Synchronization

Synchronization between running models can be implemented using events. Events are characterized by a class and a value and may be exchanged between a model and its parent model. An event queue is attached to each model to collect all events sent to this model and is managed with a FIFO policy (First In – First Out). Depending on the needs, a model may check whether its queue is empty or simply suspend its execution until it has been sent an event.

The type Event represents an event in the Mosel language. Objects of type Event may be compared with = or <> and assigned with :=. The function nullevent returns an event without class and value: this is the initial value of a newly created event and no model can send an event of this kind (i.e. the class is necessarily not null).

dropnextevent
Drop the next event in the event queue of the model.
getclass
Get the class of an event.
getfromid
Get the ID of the sender of an event.
getnextevent
Get the next event in the event queue of the model.
getvalue
Get the value associated with an event.
isqueueempty
Check whether there are events waiting in the event queue.
nullevent
Return a `null' event.
send
Send an event to a running model.
wait
Wait for an event.
waitfor
Wait for events of particular classes.

I/O drivers

The mmjobs module provides a modified version of the mem IO driver designed to be used in a multithreaded environment: memory blocks allocated by the shmem IO driver are persistent (i.e. they are not released after the model terminates) and can be used by several models. Thanks to this facility, models running concurrently may exchange data through memory by means of initialization blocks for instance.

The driver mempipe offers another communication mechanism between models: a memory pipe may be open by two models simultaneously. One of them for writing and the other one for reading. This driver also supports initialization blocks through which data is transfered in binary form.

The drivers rcmd, xsrv and xssh allow to start additional Mosel instances: they have to be used to build host specifications as expected by the connect function. Finally, thanks to the rmt driver a Mosel instance can access files available from the environment of another instance.

Driver shmem

shmem:label[/minsize[/incstep]]

The file name for this driver is a label: this is the identifier (the first character must be a letter) of the memory block. A label is not local to a particular model and remains valid after the end of the execution of the model having created it. All memory blocks are released when the module mmjobs is unloaded but a given memory block may also be deleted explicitly by calling the fdelete procedure of module mmsystem or by using the fremove C-function of the Native Interface.

Several models may open a given label at the same time and several read operations may be performed concurrently. However, writing to a memory block can be done by only one model at a time: if several models try to read and write from/to the same label, only one (it becomes the owner of the memory block) performs its IO operations for writing and the others are suspended until the owner closes its file descriptor to the specified label. Then, one of the waiting models is restarted and becomes the new owner: this process continues until all file descriptors to the label are closed.

The memory block is allocated dynamically and resized as necessary. By default the size of the memory block is increased by pages of 4 kilobytes: the optional parameter incstep may be used to change this page size (i.e. the default setting is "label/0/4k"). The special value 0 modifies the allocation policy: instead of being increased of a fixed amount, the block size is doubled. In all cases unused memory is released when the file is closed.

Driver mempipe

mempipe:name

A memory pipe is characterized by its name. Only one model may open a pipe for reading but several models may open the same pipe for writing. However, if several models try to write to the same pipe, only one (it becomes the owner of the memory pipe) performs its IO operations and the others are suspended until the owner closes its file descriptor to the specified pipe. Then, one of the waiting models is restarted and becomes the new owner: this process continues until all file descriptors to the pipe are closed.

Pipe operations are possible only if the two ends of the pipe are open: one model for reading and at least one model for writing. There is no notion of 'end of file' in a pipe: if a model tries to read from an empty pipe (i.e. no model is writing to the other end) no error is raised and the model is suspended until something is available. Similarly trying to write to a pipe for which no model is reading from the other end is a blocking operation. In order to avoid lock ups, it is usually good practice to synchronize the models using events. For instance a model waits for a specific event before trying to read from a pipe; before starting to write to the same pipe, the other model sends the expected event.

Memory pipes may be used with initialization blocks. This driver does not use labels for each record of the initialization block: it is assumed (but not checked) that both ends of the pipe are using the same sequence of records. For instance, if the writer sends an integer, a string and then an array of reals, the reader must expect an integer, a string and an array of reals: it is not allowed to skip records or change order as it is usually possible with these blocks.

Driver rcmd

rcmd:[command]

This driver starts the specified command in a new process and connects its standard input and output streams to the calling Mosel instance. The default command is "mosel -r". A typical use for this driver is to start an instance on the current machine or on a remote computer through an external program. For instance:

rcmd:rsh sunbox mosel -r

When Mosel is running in restricted mode (see Section Command line interpreter: restricted mode), the restriction NoExec disables this driver.

Driver xsrv

xsrv:hostname[(port)][/ctx[/pass]][|var=val...]

This driver connects to the host hostname running the Mosel Remote Launcher (see Section The Mosel Remote Launcher xprmsrv) through a TCP socket on port port (default value: 2513) asking for the context ctx (default: xpress) using the password pass (default: no password). Additional environment variables can be specified: assignments of the form var=val must be separated by the symbol | and variable values may include variable references noted ${varname} (expansion is performed on the remote host in the context of its environment). The special environment variable MOSEL_CWD defines the current working directory for the newly created instance.

xsrv:winbox(3211)/xpr64|MOSEL_CWD=C:\workdir|MYDATA=${MOSEL_CWD}\data

Driver xssh

xssh:hostname[(port,kwf)][/ctx[/pass]][|var=val...]

This driver is the secure version of the xsrv driver decribed above: it establishes the connection to the xprmsrv server through an encrypted SSH tunnel (using 2515 as the default TCP port number). In addition to the port number, the driver can also take a file name (kwf) used as the known host file for server authentication: this file contains the list of known hosts with their corresponding public keys. When the connection is established to the remote host, the public key stored in this file is compared with the key provided by the server. The connection is canceled if keys do not match. Generating this known hosts file requires running the command xprmsrv -key public on the remote server in order to retrieve its public key (see Section Running the xprmsrv command).

For instance, the following command will include the server mysun in the knownhosts.txt file (the command must be run on the server):

xprmsrv -key public -hn mysun >>knownhosts.txt

Then after having moved the file to the machine(s) from where connections are initiated, the following connection string may be used to open secure connections with server authentication:

xssh:mysun(knownhosts.txt)

The remote connection is handled by a separate process. By default the program xprmsrv is used as the helper program but it can be replaced by another SSH client by changing the control parameter sshcmd.

Driver rmt

rmt:[node]filename

This driver can be used with any routine expecting a physical file for accessing files on remote instances. By default, the file is located on the initial instance (node number 0) but a particular instance may be specified by prefixing the file name by its node number enclosed in square brackets. The special node number -1 designates the parent node of the current instance.

load(mi,mo,"rmt:[-1]model.bim")

In addition to physical files, this driver also emulates the behaviour of drivers cb, sysfd, tmp, shmem and java such that it can transfer streams from one instance to another. For instance, "rmt:sysfd:2" is the standard error stream on the initial instance.

The Mosel Remote Launcher xprmsrv

The xprmsrv program is the server part of the "xsrv:" and "xssh:" IO drivers: it must be running on each computer on which instances will be started using these drivers. The communication between two Mosel instances is achieved through a single TCP stream. Mosel instances are started in the context of execution environments: such an environment consists in a set of environment variables as well as the name of the program to start with its initial working directory. The server can manage different execution environments which are identified by a name and optionally protected by a password. Thanks to this feature a single server can offer several versions of Xpress or dedicated settings for particular distributed applications.

This program is also used as an SSH client by mmjob and XPRD when connecting to an xprmsrv server through a secure tunnel. Therefore it must be available when using the "xssh:" IO driver even if no server is to be run on the host machine.

Running the xprmsrv command

Main command line options

The first argument of the command that is not identified as an option is used as the name for a configuration file. The following options are accepted:

-h
Display a short help message and terminate.
-V
Display the version number and terminate.
-f
Force automatic setting of environment variable XPRESSDIR even if it is already defined.
-v [#]
Set the verbosity level of the communication protocol. The default value is 1 (report only errors) when the server is running in background (service/daemon) and 2 (report activity) when the server is run from a console.
-l fname
Set a logfile to record all messages.
-p port
Set the TCP port to listen to (default port is 2513).
-bp port
Set the UDP port for broadcast (default port is 2514).
-pf pfname
Define a file name for recording the process number of the server. This file is removed when the server exits.
-d
Start the server in background (or as a daemon on Posix systems).

The following options are used by the Windows version of the server:

-service install
Install the server as a service. All other provided options (including configuration file) are recorded and will be used by the server. If the corresponding service has already been installed, its execution settings are updated with the provided options.
-service remove
Remove the previously installed service.
-service start
Start the previously installed service.
-service stop
Stop the previously started service.
-service status
Check whether the service is already running.
-u user
This option is used only when installing the service: it selects the user running the service.
-pwd pwd
This option specifies the password required for the user indicated by the -u option.

The following options are used by all other platforms:

-u user
User that should be running the server.
-g group
Group that should be running the server.

When the server is run as a service (under Windows) or as a daemon (on Posix systems) that are usually started by a privileged user, it is recommended to use the appropriate option to run the process as an unprivileged user for security reasons. For instance, under Windows, installing the service can be done using the following command in order to use the network service account:

xprmsrv -service install -u "NT AUTHORITY\NetworkService" conffile

Similarly on a Posix system, the server can be run as the nobody user:

xprmsrv -d -u nobody conffile

Secure server

xprmsrv can also accept secure connections through SSH tunnels: this is the protocol used by the xssh IO driver. The following options are used to setup the secure server:

-sp port
Set the TCP port for SSH connections (default port is 2515).
-k fname
Private key file name.

The secure server requires a private key to authenticate itself (see following section). By default it will use the file "xprmsrv_rsa.pem" located in the same directory as the xprmsrv executable. It is important to store this file in a secure location as it identifies the server, in particular it must not be readable by Mosel models started by the server. If this file is missing or the provided file name cannot be accessed the secure server will be disabled.

Private key management

A new private key can be generated with the following command:

xprmsrv -key new

Additionally, option -k filename can be specified to change the default key file location. Note that this procedure does not remove an existing key file.

The following command loads and check the validity of a key file:

xprmsrv -key check

When executed on a valid key file this command displays the fingerprint of the public part of the key as well as its properties.

The SSH protocol makes possible authentication of a server by a client. This optional feature, supported by the IO driver xssh, requires a known host file on the client side: this text file consists in a list of host server names with their associated public key. The command xprmsrv -key public generates the required data for such a file using the hostname reported by the operating system to identify the server. Often this hostname does not correspond to the public name of the machine. In such a case, it is possible to replace the label in the file or use the option -hn name to select a different name. For instance, the following command will append to the file knownhosts.txt the public data key for the server using keyfile mykey.pem with host name srvname:

xprmsrv -key public -k mykey.pem -hn srvname >>knownhosts.txt

Mode of operation

The server proceeds as follows:

  1. If the environment variable XPRESSDIR is not defined or if the -f option is in use, the value of this environment variable is deduced from the location of the program itself. Under Posix operating systems, the environment variable XPRESS is also set up.
  2. The environment variables MOSEL_DSO (see Section Directive uses) and XPRMSRV_ACCESS (see Section Access control list) are cleared and the environment variable MOSEL_RESTR is initialised with value "NoReadNoWriteNoExecNoDBWDOnly" (see Section Command line interpreter: restricted mode).
  3. The default execution environment xpress is created: it refers to the Xpress installation detected at the first step.
  4. If available, the configuration file is read (see Section Configuration file): it can be used to define global settings (e.g., defining the logfile) or/and create and modify execution environments by defining environment variables.
  5. The process then starts its main loop listening to the specified TCP and UDP ports.
  6. When a connection is requested, a new session is started to process commands from the client. These commands are used to authenticate the client, select an environment and finally start the Mosel program in a separate process. This process inherits all the environment variables defined in the context and starts in the specified working directory (by default: the location pointed by XPRESSDIR). In addition, on Posix systems, the path ${XPRESSDIR}/lib is added to the dynamic library path of the operating system. Once the process is started, xprmsrv detaches itself from the client — the communication is established directly between the two Mosel instances.

Configuration file

The configuration file consists in a list of variable definitions of one of the following forms:

varname=value
varname?=value

Each statement is recorded in the current environment. The value may contain variable references noted ${varname}, the expansion is executed when the environment is processed except for self references that are expanded at the time of defining the variable (e.g.PATH=${PATH}:otherpath). When the first syntax is used, the variable cannot be changed by a remote host; the second syntax (using ?=) allows a remote host to modify the corresponding variable before starting the Mosel instance.

Switching to a different environment is done by giving the name of the environment enclosed in square brackets:

[newenv]

If the environment name has not yet been used, a new environment is created. Upon startup, two environments are automatically created: "global" to store general configuration and settings shared by all environments and "xpress" (it can also be referred to as * or default) the default execution environment. When the reading of the configuration file begins, the global environment is selected: in this environment all variable definitions are processed immediatly and added to the xprmsrv process environment. In this context, some variables have a special meaning and are not handled as ordinary environment variables:

LOGFILE
the file to be used for recording all messages. Messages are sent to the standard error stream when this parameter is not set.
TCP_PORT
the port number to use for TCP connections (default value: 2513).
UDP_PORT
the port number to use for UDP connections (default value: 2514). The server listen to this port for broadcast messages (see procedure findxsrvs).
SSH_PORT
the port number to use for SSH connections (default value: 2515).
KEYFILE
private key file name used by the SSH protocol (default value: xprmsrv_rsa.pem located in the same directory as the xprmsrv executable).
VERBOSITY
verbosity level for the communication protocol (default value: 1 if the server is running in background and 2 if it is run from a console).
GROUPMASK
Bit mask to select what broadcast requests to accept (default value: ANY). The server replies to a request of group grp only if bit test grp&GROUPMASK is not 0 (see procedure findxsrvs). The mask value can be given as an integer (e.g. 3 to allow groups 1 and 2), an hexadecimal number (e.g. 0xFF for groups 1 to 128) or the special keyword ANY (all groups allowed).
MAXAUTHTIME
a connection is closed if the authentication procedure takes more than the specified amount of time in seconds (default value:30).
MAXSESSIONS
maximum number of concurrent sessions (the default value 0 disables this limitation).
XPRMSRV_ACCESS
access control list (see Section Access control list).

If the corresponding command line options are used (namely options -l, -p, -bp, -sp, -k and -v) the settings of the configuration file are ignored.

In other contexts, the following variables have a special meaning:

MOSEL_CMD
the command to execute. The default value is "${XPRESSDIR}/bin/mosel -r"
MOSEL_CWD
default working directory. The default value is "${XPRESSDIR}"
PASS
password required to use this environment (empty by default). If this variable is set to the special value "*", the associated environment is disabled.
XPRMSRV_ACCESS
context specific access control list (applied after the global access list).

For instance, the following configuration file sets the logfile to "/tmp/logfile.txt"; adds the password "hardone" to the default context and defines an additional context named xptest pointing to a different installation of xpress:

# simple xprmsrv config file
LOGFILE=/tmp/logfile.txt

[xpress]
PASS=hardone

[xptest]
XPRESSDIR=/opt/xpressmp/testing
XPRESS=/opt/xpressmp/lic
MOSEL_RESTR=NoWriteNoDBNoExecWDOnly
MOSEL_CWD?=${XPRESSDIR}/workdir

Assuming the server using this configuration is running on the machine mypc, the following statements will create two instances on this machine, one for each of the defined execution environments:

 r1:=connect(m1, "xsrv:mypc/xpress/hardone")
 r2:=connect(m2, "xrsv:mypc/xptest")

Since MOSEL_CWD has been initialised with the ?= symbol, the remote host can change its working directory. For instance:

 r2:=connect(m2, "xrsv:mypc/xptest|MOSEL_CWD=/tmp")

Access control list

The environment variable XPRMSRV_ACCESS may be defined in each context of the configuration file. This variable defines which hosts are allowed to connect to the server or use a particular context. The restriction applies to the server itself when the variable is defined in the global context and as a supplementary restriction when it is included in any other context (i.e. a host cannot be allowed in a context if it is rejected by the global context).

The value of the variable must consist in a list of hosts and subnetworks separated by spaces. Each entry of this list can optionally be preceded by the + sign (for accepting the host; this is the default if no policy is specified) or - sign (to reject connection). Order of the list members is important: when checking authorisation for a given host the list is processed from left to right. The first matching entry will decide whether access is allowed or denied. A given host will be rejected if no matching entry can be found.

A host is identified by its name (e.g. myhost) or its IP address (e.g. 192.168.1.1). A subnetwork is defined by a routing prefix that can be expressed as a partial address (e.g. 192.168.1); or using the CIDR notation - the first address of the network followed by the bit-length of the prefix, separated by a slash "/" character (e.g. 192.168.1.0/24). The subnet mask may also be used instead of the bit-length which is a quad-dotted decimal representation like an address (e.g. 192.168.1.0/255.255.255.0). The special identifier ALL is replaced by the subnetwork definition 0.0.0.0/0 (any host).

In the first example below, host uranus is rejected and subnetwork 192.168.1.0/24 is allowed to connect. Note that uranus will be rejected even if it is part of the autorised subnetwork because its reference appears first in the list. In the second example, all hosts are allowed except 2 subnetworks (192.168.1.0/24 and 192.168.2.0/24):

 XPRMSRV_ACCESS=-uranus 192.168.1
 XPRMSRV_ACCESS=-192.168.1.0/255.255.255.0 -192.168.2.0/24 +ALL

All defined control lists are preprocessed just after the configuration file has been read in order to resolve host names and check for syntax errors. Unresolved host names are ignored (although a warning is displayed in such a case) but a syntax error on a control list will cause the server to abort its processing.



If you have any comments or suggestions about these pages, please send mail to support@fico.com.

© Copyright 2001-2013 Fair Isaac Corporation. All rights reserved.