wiki:docs_udpmsg

UDP messaging for HTML5-php

  • messaging is done via sending JSON encoded text via UDP
  • to *every* executable module call
    • the UDP port "_udpport" and host ip "_udphost" are given in the input JSON
    • a unique job id "_uuid" is also passed in the input json
  • to message, a UDP message is sent the the UDP port and host ip containing the JSON
    • the JSON sent is just as if you were sending at the return of the module
      • i.e. the mapping to screen elements is just the same
  • example:
    • input json to module
      {
        "_udpport" : 1234,
        "_udphost" : "some.host.ip.address",
        "_uuid"    : "1234-5678-ABCD-EFGH",
        "input1"   : "my input 1",
      ... etc
      }
      
    • typical json output
      {
        "output1"  : "my output 1",
      ...
      }
      
    • json udp output
      {
        "_uuid"    : "1234-5678-ABCD-EFGH",
        "output1"  : "my output 1 partially done",
      ...
      }
      

progress

  • adding a _progress value between 0 and 1 will provide a progress % value in the job manager
  • it is probably a good idea to fire off a udp message with a "_progress":0 at the start of the module execution
  • example:
    • json udp output
      {
        "_uuid"    : "1234-5678-ABCD-EFGH",
        "output1"  : "my output 1 partially done",
        "_progress" : 0.25,
      ...
      }
      

textarea

  • a special _textarea tag can be sent in a message or regular json
  • this will always append to a special textarea and the complete contents should be available on reattach
    • there is no implied linefeed
      • if you want to separate lines, you have to send a linefeed character
  • this textarea will only be visible when it has received output
  • example:
    • json udp output
      {
        "_uuid"    : "1234-5678-ABCD-EFGH",
        "_textarea" : "this is a message\n",
      ...
      }
      
  • note: this textarea can be controlled globally in directives.json as shown below:
      "textarea"                 : {
    # an optional label
          "label"           : "Report:"
    # the standard label defaults apply e.g.
          ,"default"        : "header2"
    # where the label will be placed in relation to the textarea
          ,"verticalalign"  : "top"
    # number of columns
          ,"cols"           : "60"
    # number of rows, if not set the textarea will autosize
          ,"rows"           : "15"
    # and of course help
          ,"help"           : "general textarea help"
    # maximum length
          ,"maxlen"         : 512000
      },
    
  • Important notes about maxlen
    • the messages sent to the default textarea are appended and if they grow too large, it can significantly effect messaging performance
    • the default value is 512000, but this can be increased up to 10000000
      • going too large is not a good idea for a system with heavy load as these are stored in the database and appended
      • a better way is to store large messages in a file and message a link to the file and only send summary data to the textarea
    • watching messages live will not truncate the messages, but reattachment will show only the truncated version
    • note this is generally a global value for a server, since only one set of messaging servers is running
      • more than one set of messaging servers can be run by specifying unique ports, but only one server can be proxy'd over http or https

files

  • returning files will not automatically get their full paths corrected under messaging, unlike normal file output types
    • this is because a message server will generally be run for multiple websites and doesn't know which site you are sending from
      • the expense of adding them all is possible, but hasn't been done, as it will require more overhead for central message processing, which we would like to minimize to optimize responsiveness under load
  • note that when returning files paths, the web browser will automatically add the correct prefix if it doesn't begin with a slash '/'.
  • input variables are provided to the executable module to ease removal of the absolute path given
    • "_webroot" generally
    • "_osroot" when running under openstack
      • these can be removed from the front of the file path along with a "/"

PERL example

Here is a perl example of an executable for messaging

#!/usr/bin/perl

use JSON;
use Data::Dumper;
use IO::Socket;

if ( !@ARGV )
{
    print "\{\"error\":\"center called with no arguments\"\}\n";
    exit;
}

$json = shift;
$ref = decode_json( $json );

# Notes:
# $ref is the input JSON decoded
# $$ref{ '_udpport' } contains the udp port
# $$ref{ '_udphost' } contains the udp host ip address
# $$ref{ '_uuid' } contain the unique job id that needs to be sent in the message

# setup up the socket handle once:

my $sock = IO::Socket::INET->new( Proto => 'udp', PeerPort => $$ref{ "_udpport" }, PeerAddr => $$ref{ "_udphost" } );


# $msg will contain the udp message text

$msg = {};

# it needs the _uuid set:

$$msg{ "_uuid" } = $$ref{ "_uuid" };

# send 15 messages once per second
# the output will display to the user mapped to fields just as in the final output

for ( $i = 0; $i < 15; ++$i )
{
    $$msg{ "completion" } = sprintf( "%.1f%%", 100.0 * $i / 15.0 );
    $$msg{ "output1" } = "$i of 15";
    $sock->send( encode_json( $msg ) );
    sleep( 1 );
}


# setup the final results
$res = {};
$$res{ "completion" } = "completed";
$$res{ "output1" } = "some output text";
$$res{ "udphost" } = $$ref{ "_udphost" };
$$res{ "udpport" } = $$ref{ "_udpport" };

print encode_json( $res ) . "\n";

PYTHON example

Here is an example method using python. Note that the module program that runs sends back a text string (via txtOutput) that contains a keyword with the fraction of the completed run.

   import locale,socket,multiprocessing

   def background_job(self,process,txtQueue,json_variables):

        total_string = ''

        UDP_IP = json_variables['_udphost']
        UDP_PORT = json_variables['_udpport']
        sock = socket.socket(socket.AF_INET, # Internet
                socket.SOCK_DGRAM) # UDP

        socket_dict={}
        socket_dict['_uuid'] = json_variables['_uuid']

        while process.is_alive():
                try:
                        this_text = txtQueue.get(True, timeout=0.1)
                        text_split=string.split(this_text)
                        if(text_split[0]=='STATUS'):
                                value = locale.atof(text_split[1])
                                svalue = str(100*value)
                                socket_dict['_progress'] = value
                                socket_dict['progress_output'] = value
                                socket_dict['progress_html'] = '<center>'+svalue+'</center>'
                                doc_string = json.dumps(socket_dict)
                                sock.sendto(doc_string,(UDP_IP,UDP_PORT))
                        else:
                                total_string += this_text
                except:
                        if not process.is_alive():

                                #doc_string = json.dumps(socket_dict)
                                #sock.sendto(doc_string,(UDP_IP,UDP_PORT))
                                return total_string

                time.sleep(0.01)
        else:
                return total_string

        return total_string


reattachment and messaging

  • The last message for a unique global job id is stored when received by the php websocket daemon
    • the message is stored in a global msgs database keyed by the unique global job id
  • when 'reattaching' to a job via the job manager, the last stored message is sent to the client
  • Due to event sequencing (below), if the udp message from the executable is received by the php websocket daemon after sequence step 1. but before sequence step 2. the client will receive the message twice
    1. subscribe to messages for a job id
    2. start result retrieval watchdog (which will start with the last stored udp message if available).
Last modified 15 months ago Last modified on Nov 29, 2016, 3:28:39 PM