player and editor for (semi) augmented reality interactive movies and games (for Android and iOS)

The project consists of a player and an editor application for the Android and iOS platforms. Briefly it should make the following possible: People can use the editor application to record movie sessions, for which every frame is stored together with the orientation/angle data of the mobile sensors. The editor will also be used to add location data to the recorded sessions. This will work by editing a trail on a virtual map along the timeline of the recorded sessions. This trail defines the possible paths to be replayed later by the player. Unfortunately GPS data is not sufficient, because the applications should also work in indoor locations where GPS is not available. That's why manual editing will be required. This virtual map can also be displayed on the player as a convenience feature. This should make it possible for the player application to replay the recorded movie data, but this time interactively by allowing the user to control the orientation and select paths to take (which is basically replaying previously recorded session data, but this time interactively and conditional). The possible angles/orientation and moving paths are limited of course by the available recorded data. The controls will show what possible actions are possible at any time. Read more under 'Detailed requirements'..... **Continued under 'Detailed requirements' below:** ## Deliverables This editor will also be used to combine the different recorded sessions together and define the possible transitions between them. In the end that will result in a definition of possible paths (which can be conditional branches, see below) between the recorded sessions. Also, for every scene or transition, it's possible to specify whether the recorded data will be played automatically without any possible user intervention, or purely driven only by user interaction. At certain points during playing, it's also possible to define that a user choice or intervention is required to proceed (which determines the next sequence to be played). The interactive (and gaming) elements of the project consists of the fact that it's possible to define custom elements in the recorded data (by using the editor), which is also the semi augmented reality part of the project. The recorded sessions can be enhanced by virtual elements/objects over the real data, which can be selected later in the player as part of the interactive experience. To make this possible there should be a scripting language which can make use of these defined virtual objects, so that conditional paths and sequences during the player session become possible. This scripting language can also manipulate state and possible actions of other virtual objects. Overall the project should make it possible for users to create interactive movies and games. Think of the resulting games like point and click adventure games, but now in recorded movies enhanced by a virtual layer. Users can then share these movies and games to other users, so some kind of community/repository should be available in the player application. For a start this could be as simple as an (searchable) overview of available movies/games, where users can add comments and a simple rating. When a user has created a game or movie, he/she can publish it to this central repository. My idea is that overlaying a gaming engine as a virtual layer over the recorded movie sessions should make this possible. I believe the Unity engine is the best option currently. But any other ideas are very welcome.