Media Block Paper Research Notes

tracked windows

done for not distributed operation.

How should it work with the distribution ? work this out with locales ?

current problems :

• window parameters are shared because windows also are referenced from within the context kit. the contextkit itself is distributed. it is possible to move the window with the pen, and this is shared. however there seems to be a bug somewhere because setting the windows position / orientation via accessor methods does not work !
• current implementation relies on getting the window at the right position etc.
• current implementation relies on geometry that is not shared, because it is not in a window.
• appearantly only one window can be maximized at any times. this window is behind the normal windows in the scenegraph and therefore any 3D events are first culled to the normal windows :(. we would need a way to share additional geometry. the idea could be to have some additional geometry in the context kit (this should be possible) and then it is shared automatically !!!

The following implementation should work :

1. Fixed. these are fixed to a certain window of a certain context (or application ?). this is defined by setting context name and window number as parameters. another parameter is then whether the context should actually be started or not, if the window is not present when the marker is tracked.
2. Free. Free markers can be helpers to organise your environment. they attach to a window close by and manipulate it from then on. releasing a window is possible by turning them over (or some other gesture).

To share information make a context that keeps the marker geometry as children and puts it also into the auxRoot. only the master has necessary callbacks (if there are any). the set window is found by its name and the name is shared across. window position itself is then set individually (otherwise it doesn't work anyway).

carrying an application over with a marker between to not shared locales.

• offline variant : as soon as marker leaves locale L1, the app is safed under a well known place, and reloaded into L2 as soon as the marker appears there.
• online variant : the app continues to run in L1, but is distributed to L2 as soon as marker enters there.

the typical media block steps sound like :

1. a marker appears and an app is started from somewhere. this can include an app that was saved there before hand, or was there already forever.
2. a marker triggers sharing of an application that was not shared before.
3. a marker just moves an application that is present.
4. vanishing of a marker triggers saving the application

in all cases applications are started by markers ?

application start

done for not distributed operation. it uses simply the SoLinkApp node.

How does it work with DIV ?

Not good, SoLinkApp node appears to start stuff at the slave as well (confirm that ?), would be simple to change in the SoLinkApp node with the typical DIV code...

even better only do it with Zaajic enhancements.

semantics regarding restarting of applications is not clear.

locale support

working on design for implementation...

• a locale node with a location. this is implemented as a nodekit derived from SoTrackedArtifact.
• two different children types are supported :
  • userkits - each userkit is assigned to a locale to be able to interpret its tracking data correctly.
  • contextkits - each contextkit is also in exactly one locale (per host - not necessarily within the whole system)
• with the rules laid down by Dieter any context can exist in only one locale at each host, therefore it is not necessary to come up with some additional store of children position per locale etc.
• on the other hand this information has to reside somewhere as well, possibly within the locale ? ( we don't do this currently - here we might have to come up with some solution to not share a window position :)
• Tracking event data is transformed according the locale it is assigned to. its location information is transformed with the inverse of the locales location and then send to all. everyone using it needs to transform it along the path from the root, therefore in the defining locale it will be in the correct coordinate system again.
• However track engines output (global fields) is not transformed. this is because it is used by objects that already interpret it within their local coordinate system without having to establish the world to locale coordinate transformations.
• camera is simple as it is within a locale as well. still have to come up with something to include the full world to locale coordinate transform :(.
• location information of locales could be distributed using DIV as well. on a level of the locales and exluding the actual window scene graphs below.- don't know if this is feasable - anyway it is not implemented that way now.

dynamic workgroup management

right now everything has to be configured at startup and all applications are shared by default (except certain default system applications).

this should be changed to include the following properties :

• for each application we want to decide whether it should be shared or not
• this status should be changeable ? yes or not
• there could be several sessions a host is participating

some requirements:

• to make div etc. work, we need the same user environment, therefore some kind of user exchange must happen !
• exchanging userkits requires that uids are unique on all machines (why not)
  use zaa usermanager ! maby we should use sessionmanager for user information as well (as it is there already)
• need to be able to share already running applications. this is like streaming data to a newly connecting machine, except that it now happens to the session manager as well.
  Zaa integrieren. using zaa app loading we should be able to pack a context in a file again and
  stream it to the other programs etc.
• differentiate applications with respect to sharing make it possible to select for each application whether it is shared, if possible. this could be integrated with the locales
• differentiate whether host will participate for a certain application. this could be integrated with the locales
• simple way to extend Gerd's work :
  advertising for session manager, include session manager in studierstube, dynamic connection to session manager, etc.
• dynamic connection to session manager is necessary for adhoc networking (by definition :)

the current idea is to use one session manager, that all hosts are permanently connected to. the session manager knows all locales and stores information about them and the contained contexts. each host subscribes locales it is interested in. all applications within a locale are shared if the locale is shared. therefore all local applications should reside in a special local locale that is not shared !

this contradicts one or two things said so far : the spontaneous joining (not necessarily, this can also mean join a locale), but definitly the adhoc networking. as far as i understand adhoc networking means establishing a network connection (layer 1) adhoc. therefore a permanent connection to a common session manager would not be possible.

on the other hand the following interpretation is possible : contact with a common session manager is permant but via a low bandwidth network. collaboration is only possible via high bandwidth networks, which would be established adhoc.

user management design - new

The new user management follows the following design :

assumptions :

• the users managed (i.e. rendered etc.) by the local process are configured in the local userkit file. These are read at startup and added to a locale.
• during runtime only remote users are added and removed by joining or leaving a locale.

some problems, that influence the design :

• the userkit scene graphs as used at runtime are not useable for distribution. we have to use the 'pure' versions found in the user kit file.

To distinguish between local and remote users we introduce a new flag 'local' to designate a user as local or remote. This flag is only managed by the distribution manager. The following events can occur :

• startup : the local userkits are read from the userkit file and registered with the user manager. they are added to a locale. because this is during startup the locale is not shared yet.
• the userkits are also passed in pure form to the distribution manager to store for later.
• sharing a locale the distribution manager iterates through the userkits already present in the locale. These are by default local and are flagged appropriately. using the userids they are retrieved in pure form from the storage and send to the distribution manager.
• any remote users are received from the session manager and flagged as remote. then they are registered and added to the locale.
• upon removal of remote users, they are deregistered and removed.
• leaving a locale, the distribution manager iterates over the userkits in the locale. any kits flagged as remote are removed. for all kits flagged as local the session manager is notified that these kits are removed.

actions for sman

• subscribing locale
  • get users in locale, this gets also all devices they use such as pip sheets and pens etc.1)
  • add own users to the locale, if not present etc.1)
  • get all current apps
• leaving a locale
  • remove subscription to apps
  • remove own user from locale.1)
  • remove unnecessary user kits etc.
• starting a new application in a locale
• closing an application. this needs to be done via sman now, because for an application in two invisible locales there are no necessary tracking infos shared to trigger closing via local mechanisms
• changing master, this could be implemented via DIV, probably ??

1. actually each userkit needs to appear only in one visible locale subscribed by the host. however it or its devices need to appear in at least on locale ! this is not necessarily happening by default !

problems

with distribution:
numerous crash conditions, appearantly due to missing pips, race conditions etc. just try to make it 'fail-safe' by ignoring most error conditions.

demos - new

single mobile user

a simple demonstration for locales for tracking purposes. we use a default locale A for a body stabilized system and a second locale B for a orientationally world stabilized system. The default locale A contains the userkit and all tracking is done directly using the ARToolkit tracking. The locale B is additionaly tracked using the InterTrax. Some applications are started in locale A, others in locale B. This will be hardcoded, as we have no user interface or configuration mechanism yet for these things.

dual mobile user

two mobile users working on one application. there are two different locale setups possible :

1. shared locale for the application

   each user has its default locale A and A' containing their respective userkits. Tracking is done again via ARToolkit.

   Moreover the application itself is contained in a shared locale B. The locale is tracked using a marker and the application is in the origin (thus it will appear as attached to the marker).

   Both hosts subscribe all locales. the locale of the other host A' or A respectively are tracked using the marker as well. Thus the locations of all userkits are transformed into the hosts default locale and distributed processing is possible.

2. application replicated in two locales

   Both users have each a default locale A and A' respectively. The userkits are located in these locales.

   Both hosts start or share the application in their default locale without sharing a locale. Thus the application is replicated or shared over two locales 1). In this scenario only one user can interact because the tracking data is not distributed to all hosts. however by setting the focus policy to click for master, we should be able to implement fast and almost transparent master and input switching.

stationary setup + mobile user

spontaneous collaboration with stationary setup. this is the 'walk in' scenario.

We configure the two stationary users to work in a single locale A. Within the locale we have a tracked marker (overhead camera) that is used as a reference point for other locales that are attached to tracking cameras (inside out tracking of the mobile setup). They can start any application they like (as long as it is ported to the new studierstube :).

A mobile setup is configured with its own default locale B. This locale contains its userkit and tracking is done only using the ARToolkit camera tracking.

As soon as the reference marker is detected by the mobile setup it joins locale A. (somehow the other hosts join locale B 2)). It tracks the locale A via the reference marker position and its own measurement of it. The stationary host does the same in the reverse direction. all applications are shared by the mobile setup now as well...

A 2 mobile user version of this scenario is the 'spontaneous collaboration' scenario. Technically, the former and the later should be feasible, if one works.

sharing applications between different locales

Here we would use locales to organise the sharing of an application in two very different setups / situations etc.

collaboration of stationary and desktop setup. good demo to show locales concept. two widely differing setups and user interface paradigms would interact in the same work space. We statically define two locales and share an application. this should be a debug setup reused in an actual working setup...

viewing an active application on a projection setup. a stationary user is working on an application while another user is observing this on a projection wall handling the application only with a tracked marker.

use of mediablocks

mediablocks are used in a number of ways :

• starting applications - as soon as a marker is found a new application is loaded (an shared if in an appropriate locale)
• sharing applications - as soon as a marker is found an application is subscribed and shared too 1).
• moving application windows - a window can be permanently or temporarily be manipulated by a marker
• joining locales - as soon as a marker is found a given locale is joined. this may trigger joining more locales and sharing applications as well.

1. This requires support by the distribution manager to trigger application start at the client without loading the app. the session manager should already support all that is needed.
2. Having already joined hosts join the locales of a new host requires knowing them about what locales the new host is in. This operation effectively computes the closure of locales of a set of hosts sharing a locale. It may trigger a cascade of joining events as the new host has to join more locales as a consequence of other hosts joining its old locale...

log - next steps and what is done - as of 3-06-2002

Plan:

• include distribution fully in stb
• solution for cameras within tracked locales...
• add monitor to session manager
• add possibility to have several instances of an app ?
• fix any crashes by simply ignoring most faults

Done :

• add compression to div
• fix some missing events regarding pip sheet stuff
• tests with multiple locales
• dynamic user management - requires sharing of user information - zaa
• finish sman2
• test to add userkit
• integrate zaa work
• div moved to ACE
• distrmanager moved to ACE
• improved ACE reliable multicast, almost as good as div
• checked and tested zaa work, but not on modified version.