Instead of a virtually controlled camera via a 3D modeling package, we also plan on investigating a motion capture camera using the Vicon system in the robot lab. We will construct a 16:9 frame approximating the view of the camera that the director can carry around the scene. Three tracking spheres on the frame will have line-of-sight contact to the tracking system and will be recorded to 3D points over time. After some basic smoothing and throwing out erroneous data, the string of points will be converted into a curve for translation and further refinement for recording. This step breaks us free from 3D modeling as a basis of the video, and allows for haptic and hand-held control of the camera, while offering precise repeatability.
- Mo-Cap controlled Focal Lengths
The second level of complexity of motion capture still utilizes the tracked frame, but includes another point – a sphere tracker on a telescoping pole. This enables the director (or likely an assistant) to dynamically place the sphere on the object of focus, and after recording the frame and focus sphere simultaneous, recording the scene using the follow focus from Step 2.
- VFX matched into Robot Environment
Beyond controlling the camera for recording, we are also looking into adding digital visual effects in post-production over the recorded video. This becomes trivial as we already have a digital representation of the scene and camera positional information. Often without this data, VFX companies use matchmoving or structure from motion, a computer vision technique that backsolves for the position and orientation of the camera by analysing and comparing multiple frames from the sequence.
We intend to animate some simple digital objects in the scene to matte over the footage and show a proof-of-concept that 3D animation is easily overlaid onto robotically-recorded video footage.
- VFX match over other Robot
The last technical hurdle we intend to vault is animating moving objects in the scene while the camera is moving. Much like the previous step, it involves compositing 3D animation, but now this 3D animation is moving, likely with or because of the other robot.
- Final Film linking together all of these
The eventual result of these experiments will culminate in a short film showing of the individual technique and virtuosity of each. We plan on collaborating with a film student to generate ideas on content and story, likely focusing on the robot lab space and tools inside it.