NETRPC
======

What found here makes RTAI a distributed system, both for kernel and user 
space applications. The use of "net" in front of "rpc", for the name of this 
directory main module, is due to the existence of rpc functions internally 
to all RTAI schedulers, the concept of intertask Remote Procedure Call (RPC)
messaging being strongly supported within RTAI since its inception. Clearly 
nothing new, synchronous intertask message passing is an old concept and the 
basis of microkernels, either distributed or not. So this implementation is 
nothing but the RTAI specific way of doing it.

The basic specifications for making it easy are:

- use just the name of any already available function substituting "rt_..." 
  with "RT_...",
- add two initial arguments more, i.e. the "node" of and the "port" on the 
  remote machine that will execute "rt_..." functions that became "RT_...".

e.g.: 
	  rt_mbx_send(mbx, msg, msglen);
becomes: 
	RT_mbx_send(node, port, mbx, msg, msglen);.

Using 0 for the "node" forces a local execution of the corresponding "rt_..."
function. In this way you revert to a standard local application automatically.
"Port" is not used with a zero "node" but should be kept to a meaningfull value 
anyhow, to allow remote calls when "node" is set to point to a valid remote
machine again. So using "RT_..." naming with a null variable "node" allows a 
local application to be ready for becoming distributed, once you reassign a 
real "node". The only added cost will be a longer arguments list and the 
execution of a "C" "if" instruction. The "node" and "port" arguments will 
receive further attention shortly afterward, i.e. when we'll explain how to
request "port"s on "node"s.

Naturally you can also run a distributed application wholly in local mode by
just setting all your "node"s to the local host (dotted decimal notation: 
127.0.0.1); less effective than using a null "node" but useful for testing 
its networked execution on a single machine.

The only change in the formal arguments, with respect to the usual normal local 
"rt_..." usage, is related to any possible argument associated with a time 
value. It must be expressed in nanosecs always, instead of the RTAI standard 
internal timer units count. In this way you are not required to know the 
frequency of any timer running on a remote machine and be sure that a correct 
timing will be guaranteed anyhow, irrespective of the node on which it will run.

These are, more or less, the only things to know to make any RTAI application 
distributed. 

There is however a possible usage variation obtained by using "-port" instead
of "port" in any call. In fact a "port" is a positive number and making it 
negative forces an immediate return, without waiting for any result, i.e an 
asynchronous rpc. It is your responsability to ensure that an asynchronous rpc 
is reasonable, e.g. a "sem_signal" can be used in such a way, while a "sem_wait"
will likely cause missing a possible synchronization. Notice that the remote 
support stub will answer anyhow but the returned values will be discarded. 
So it should never be used if you want to know the result of your remote call. 
Nonetheless, as it will be explained later on, a mechanism is provided to allow 
recovering the results of any asynchronous call left behind.

In any case it is important to know that any further asynchronous call will 
not be sent if the previous one has not been answered yet. "Net_rpc" assumes no 
buffering at the remote node, i.e. one can be sure the remote port is ready
to accept calls only if its stub has answered to any previous service request 
anyhow. So, if the previous one was not answered yet, launching an asynchronous 
call will cause an immediate return without any action being taken. In relation 
to this it is important to note that the overall maximum message length 
allowed, both in sending and receiving is the MAX_MSG_SIZE found in netrpc.h, 
actually 1500 bytes, i.e. the number of octets defined by Linux ETH_DATA_LEN.

The user can check if the port has or has not answered an asynchronous RPC by 
calling:
	void *rt_waiting_return(unsigned long node, int port);
a non null return value implying "port" at "node" is waiting an rpc return.
Remote calls that make sense being used asynchronously can be used in
interrupt handlers also, see directory "resumefromintr" for an example.

Before doing anything remotely a task working on remote services has to ask 
for a "port" at its remote peer. Such a port is obtained by a call of the type: 
	myport = rt_request_soft_port(node); for a soft real time service 
or
	myport = rt_request_hard_port(node); for a hard real time service 
and released by:
	rt_release(node, myport);
when the "port" is needed nomore. A task cannot ask for more then one "port",
multiple "port" requests from a task will always return the same "port". The
assigned "port" will be an integer >= MAX_STUBS, defined in netrpc.h, it is
related internally to a socket and port but its value as no reference 
whatsoever to either of them. A "port" carries a couple of informations in it,
i.e.: an index in the remote list of port; the type, 32/64 bits, of remote
machine.

Nonetheless a task can create and access more "port"s by using: 
	anotherport = rt_request_soft_port_id(node, id);
	anotherport = rt_request_hard_port_id(node, id);
"id" being a unique unsigned long "id" the task must have agreed with any
other local application. Releasing a "port" defined with a specific "id" 
makes no difference with the simpler request above, i.e.:
	rt_release(node, anotherport);
must be used anyhow.
Multiple ports per task can be used by an application to implement some sort
of buffered requests, whose detailed implementation is at a user's taste. So
even if NETRPC does not provide fully asynchronous buffered APIs it makes
available the tool for a relatively easy implementation of such an rpc policy.
It nonetheless important to remark that the actual implementation of "netrpc"
without buffering requests is a design choice. Buffering would be trivial to
implement by using RTAI mailboxes in place of semaphores, to synchronize
remote stubs, but real time require synchronization to let one know if 
everything is marching appropriately. Too many things buffered are doomed to
be related to missing deadlines.

In requesting a "port" there is also the possibility of providing a mailbox 
to recover results of asyncrhonous calls. So you can use either:
	myport = rt_request_soft_port_mbx(node, mbx);
	myport = rt_request_hard_port_mbx(node, mbx);
or:
	myport = rt_request_soft_port_id_mbx(node, id, mbx);
	myport = rt_request_hard_port_id_mbx(node, id, mbx);
"mbx" being a pointer to a mailbox. When a new rpc is made and there is the
result of any previous asynchronous call pending it will be sent to such a
mailbox. A typical use of this possibility consists in providing a server
task that reads the mailbox and uses the returned values in a manner agreed
with the original RPCs sender. A more direct way to ensure a pending return
is sent to the mailbox is to use:
	int rt_sync_net_rpc(unsigned long node, int port); 
which forces a synchronization, thus sending any pending return to a mailbox, 
if one is made available at the port request, it returns 1 always. Such a
function allows to recover a pending return immediately. It is likely it
will be used in combination with "rt_waiting_return". 

A helper functions is provided to obtain any result queued in the mailbox:
	int rt_get_net_rpc_ret(
		MBX *mbx, 
		unsigned long long *retval, 
		void *msg1, 
		int *msglen1, 
		void *msg2, 
		int *msglen2, RTIME timeout, 
		int type
	);
mbx:		The mailbox
retval:		The value returned by the async call, if any. A double long 
		can contain any value returned by RTAI functions, it is up to
		you to use it properly.
msg1 and msg2:	Buffers for possibly returned messages.
msglen1, 
msglen2:	The length of msg1 and msg2, the helper function return the 
		actual length of the messages, truncating them to msglen1 and 
		msglen2 if their buffers are not larger enough to contain the
		whole returned messages.
timeout:	any timeout value to be used if needed by the mbx_receive 
type:		defined by type, the mailbox receive function to be
		used, i.e.: NET_MBX_RECEIVE for rt_mbx_receive, 
		NET_MBX_RECEIVE_WP for rt_mbx_receive_wp, NET_MBX_RECEIVE_IF 
		for rt_mbx_receive_if, NET_MBX_RECEIVE_UNTIL for 
		rt_mbx_receive_until and NET_MBX_RECEIVE_TIMED
		for rt_mbx_receive_timed.

This function is just a helper, a user can see it as a suggestion for his/her 
own optimised implementation, e.g getting just the returned value or a single 
message, because s/he knows those are the only returned values. See the test 
"uasync" for a specific example.
 
It is stressed again that even such an asynchronous form of "netrpc" does not 
queue messages, as said it allows just one effective async call, but can 
help in increasing the application parallelism without loosing determinism. 

"Port"s requests cause a task rescheduling, to wait for the answer, and have
an owner that is identified by combining the requesting task handle, or "id",
and the node identifier, so they should be called just from within an RTAI 
task. Thus you have to care setting up an initializiation task if you want 
to use them in a "make it all at once" unified initialisation. Better never 
call them directly in init_module functions, in case of kernel space 
applications, even if you use a reqiest port with "id".

"Port"s request/release need to be timedout, and possibly repeated, in case of 
collisions on the unique server "port", preset and used by "netrpc" to accept 
the related requests on the remote node. A Linux timer is used for this scope,
since it is possible that no RTAI timer is running at the time a request is 
made. So "port"s request/release are not real time operations, they must be 
carried out before beginning any true real time work. Because of this assumption
they are not time critical and are timed out softly, without too much a hurry.

A function is provided to set up stub response timeout on each port. By default
each calling is set not timeouted. Setting timeout on port is obtained by
calling:
	rt_set_netrpc_timeout(port, timeout);
Each task using this port after calling this function waits stub's response
for "timeout" period and then return RTE_NETIMOUT error code.
If you want removing timeout on "port" simple call:
	rt_set_netrpc_timeout(port, 0);

If for some reason on remote nodes arrive some request on the wrong port a new
port is dinamically assigned and an error RTE_CHGPORTOK is returned. The 
developer must check integrity on remote node, when the error is found. If in
this situation a new dinamically assigned port is not available RTE_CHGPORTERR
error is returned.

A further remark must be made in relation to intertask receive functions.
If you want to receive from any task there is no need to use RT_receive(x),
rt_receive(x) will receive also from remote tasks. If you want a specific
receive instead you must use RT_receive(x), recalling that you will receive 
from a local stub acting as the agent of the remote task. RT_receive(x) takes 
care of it, with a little added overhead since it has to find the stub 
associated to the node-port-task appearing in the RT_receive(x) argument list. 
You can improve the efficiency of specific receives from remote tasks by 
finding their local stubs yourself, using:
	RT_TASK *rt_find_asgn_stub(unsigned long long owner, int asgn);
where "owner" combines the remote "node" and "task". It can be built by using 
the macro: OWNER(node, task); "asgn" will assign a port-stub combination to 
the owner pair if it is different from zero and none exists yet. Any following
request for a "port" from the remote "task" will be given the one you have
created. With such a technique you can then effectively use a specific
rt_receive(x) on the task returned by rt_find_asgn_stub. The gain will be
more significant for user space applications, even if it is likely that you
will not notice any difference. 
See the 3 mode of executing found in "usoundmesg" for an example of what just
explained.

The same reasoning applies to RT_return(x). Take care of using RT_return(x)
with RT_receive(x) and rt_return with rt_receive(x), never mix them.  

Notice that "node" must be the networkwise integer corresponding to the
dotted decimal notation of the remote IP address. In kernel space the function 
"ddn2nl" is provided to transform the standard dotted decimal notation, 
xxx.xxx.xxx.xxx, into such an integer. In user space standard libc functions 
can be used directly, see the examples. 

The "ddn2l" prototype is:
	unsigned long ddn2nl(const char *ddn);
a null value being returned for a bad "ddn" string. A "ddn" string is assumed
to be bad if any of its dotted field contains a value greater than 0xFF.

The netrpc driver module can be made to know its local node by being insmoded 
either with the parameter <ThisNode="xxx.xxx.xxx.xxx"> or by a call to: 
	rt_set_this_node(const char *ddn, unsigned long node, int hard);
made by the application module before using any netrpc service. In the above 
call "ddn" is the dotted decimal notation string and node the corresponding 
"node". A call with a NULL "ddn" means that "node" contains a valid value while
a non NULL "ddn" implies that the local node is obtained from converting the 
related string, "node" being discarded.
In case of RTnet is enabled rt_set_this_node must be called twice. One for each
mode, hard and soft, to set up corresponding IP address. The other way doing
that is insmoding netrpc with <ThisHardNode="xxx.xxx.xxx.xxx"
ThisSoftNode="xxx.xxx.xxx.xxx"> parameter.

"Netrpc" basically uses Linux ethernet support. In this case communication are
not RealTime. Installing RTNet (http://www.rts.uni-hannover.de/rtnet/) enable
Netrpc providing true real time communications.
As already hinted "Netrpc" can serve soft and hard ports alike, the difference
being related to the use of a soft real time stub task, for a soft port, while
a hard real time stub is used for hard ports. In the soft case "netrpc"
emulates RTNet internally using Linux Ethernet support, so it does not supply
a real time networking. To use NETRPC communications in true hard real time
you must install the real RTNet services and inform "netrpc.c" that it is
available, by setting the related option in RTAI Menuconfig.
When both support are enabled calling all function must be done using soft node
IP, hard IP is internally menaged, for this purpose is necessary to set them up 
insmoding netrpc or using function as explained above.
When there is only soft support in place netrpc.c automatically sets network
calls appropriately. When both are used it is under user responsability to
specifically ask for the preferred service. The choice comes through using 
the port_request functions, already described above.
It is important to notice that a soft port requests implies a remote soft stub
support, while a hard port request defaults to a hard stub. While it can make
sense using real time networking in soft real time tasks the opposite is 
likely mostly inappropriate. So "netrpc" forces the stubs to be soft when just
Linux networking is available. 
However there is no constraint imposed by NETRPC in the case both supports are 
in place and the user can choose whatever s/he wants.
Notice also that RTNet now imposes that hard communications be done from hard 
real time tasks only. It is thus compulsory to adhere to this condition in
developing your own applications.

It must be noted also that the specific implementation adopted for our RPC
scheme acts along the ideas already used for LXRT. In fact, networking
apart, the remote execution is carried out by a stub task, you can call it
buddy, proxy, agent, already hinted at. The possibility of expanding its use 
to any user specific application is already in place, once more following the 
ideas used in extending LXRT.

Netrpc has no security and error correction in its code but let its user
implement its own by hooking into it encode/decode functions and APIs table 
extensions. The encode/decode functions can be used to provide data 
compression and error checking also, even in a selective way by assigning 
ids at port requests.
Since such a service can be costly for real time applications it has been
preferred to avoid any internally coded scheme. In fact real time networking
will likely be mostly used on trusted "internal" connections requiring none, 
or a very simple data encoding at most. Thus it is better to let a user do 
what s/he wants. A simple demonstration of encode/decode is provided in 
"showroom", see "share/scrambler". It contains also an example of a user
specific NETRPC calls expansion.

To verify almost all the possibility offered by "netrpc" there are:
- an example in kernel space (basetest),
- five examples in user space (basetest, sound, soundmsg, resumefromintr, 
  async). 
Anything that works in user land will work in the kernel land also. That's why 
there is only one example in kernel space. 
"Netrpc" is now a production tool. In fact RTAI has a production application, 
i.e. RTAI-Lab, that is fully based on "netrpc". At the moment this text is the 
only documentation available. You should look also at netrpc.h, the above 
cited examples and RTAI-Lab to better understand what can be done with it. 
So the tool for integrated symmetricall local/distributed-kernel/user-space 
application is available and its usage is up to you.

For your convenience a list of the functions and macros explained in this README
follows. There is clearly no need to explain "RT_..." services since they 
follow the general rules explained at the very beginning.

- int rt_send_req_rel_port(unsigned long node, int port, unsigned long id, MBX *mbx);

- #define rt_request_soft_port(node)           rt_send_req_rel_port(node, 0, 0, 0, 0)
- #define rt_request_soft_port_id(node, id)    rt_send_req_rel_port(node, 0, id, 0, 0)
- #define rt_request_soft_port_mbx(node, mbx)  rt_send_req_rel_port(node, 0, 0, mbx, 0)
- #define rt_request_soft_port_id_mbx(node, id, mbx)  rt_send_req_rel_port(node, 0, id, mbx, 0)

- All of the above in the hard form, e.g:
- #define rt_request_hard_port(node)      rt_send_req_rel_port(node, 0, 0, 0, 1)
and combinations thereof.
For backward compatibility reasons there is a version also without hard_soft, 
e.g. simply rt_request_port. It defaults to a soft request.

- #define rt_release_port(node, port)  rt_send_req_rel_port(node, port, 0, 0)

- unsigned long ddn2nl(const char *ddn);

- unsigned long rt_set_this_node(const char *ddn, unsigned long node);

#define OWNER(node, task) \
        ((((unsigned long long)(node)) << 32) | (unsigned long)(task))

- RT_TASK *rt_find_asgn_stub(unsigned long long owner, int asgn);

- int rt_rel_stub(unsigned long long owner);

- int rt_waiting_return(unsigned long node, int port);

- int rt_sync_net_rpc(unsigned long node, int port);

- rt_get_net_rpc_ret(
	MBX *mbx, 
	unsigned long long *retval, 
	void *msg1, 
	int *msglen1, 
	void *msg2, 
	int *msglen2, 
	RTIME timeout, 
	int type
  );

As usual ... comments and bugs fixes are welcomed.

Paolo.
