Virtual Production
Virtual production is where digital and real world elements combine at image capture. So examples of virtual production are rear projection, motion capture, and also using an LED Volume.
Often times using LED walls will be better than rear projection because of increased contrast, brightness, control.
Poor Mans Process
Poor Mans Process is when you shoot a car on a stage or location to make it appear as if it is moving. This is accomplished with moving lights, pushing on the suspension of the car, and potentially even adding atmosphere into the scene, like rain.
Lighting
- Lighting Movement - Panning Lights
Pan lights on stands, turning them off as you pan past, then turning them back while they are off. This method can be great for slower moves, but can be really difficult for faster movement or when rapid panning is necessary. If you would like a faster movement, battery powered lights are recommended to prevent the cable from being tangled.
- Programming chases
You can also use LED lights or even tungsten lights and pixel map them to create movement by using a chase on a lighting console. This can be good as it doesn't require any technician to be involved and it's extremely repeatable.
- LED Walls
Also called Rich Mans Process. Can use LED walls and put video on them from a media server to use the plates to light the scene. Much more accurate lighting than the others and can also provide you with reflections if you put one over the hood and windshield.
Other Movement
- Camera Movement
Should be shot with a loose head so you can feel some movement of driving in the vehicle.
- Car Movement
Grips will either put 2x4 pieces of lumber into the suspension of the car to rock it back and forth, or they could just be on the bumper if they are out of the shot. A small rocking can be convincing that the car is moving.
LED Volume
The main reasons to shoot on a volume are because you need to shoot on Impossible Locations, like Mars, or something very stylized. It could also be because of Logistics and Schedule.
You should not shoot on a volume if:
- You need to light with Hard Light or do Day Exteriors. Very difficult to do day exterior because you need the light to be very far away and very hard.
- If you are doing simple sets, it is easier to just build it. Shooting a small apartment for example is far easier to build than doing a volume.
- Lack of lead time. It can take 8-12 weeks to build an environment. You are shifting post budget to prepro to build these environments.
Ultimately the DP should help decide when and if to shoot on a volume.
Design
When doing virtual production with LED walls there will be a virtual art department in addition to a real art department. They will have to generate a number of assets, this would be props and set dressing that might have to exist both in the physical and the virtual environment, amongst other things. You will need to scout as well which can be done via zoom or using 3d goggles.
Asset Generation can be done by:
- CGI - Making an item purely using computer graphics.
- Photogrammetry - where you take a ton of stills of an item, then import those into a computer, then have the computer create a 3d model.
- LiDAR scanning - where you scan the geometry of an item, then texture them with photographs of the item along with marketplaces and buying pre-made assets.
- Items that may be in the set and used in the foreground must be made 4 weeks out so that there is time to scan it and also put it into the virtual enviornment.
Scanning real items is always going to be preferable.
Camera Tracking
The camera must be tracked in the space so that the LED walls know what to display and how much to move the background in relation to whatever you have in the foreground.
Outside in Tracking
Sends light down to the stage, the light gets bounced back off a reflective ball on the camera. You can also track other items besides the camera too with this system. An example of this is a flashlight, because the computer can track where the flashlight is, they can have the actor interact with the set and you’ll end up with light on the background.
Inside Out Tracking
Light on top of the camera, sends light to the ceiling and there are receivers on the ceiling that can help figure out where the camera is. Easier to place more receivers on the wall, harder to add more using the other system. Can even place the constellation directly on diffusion if you have overhead lighting.
LED Wall Specifications
Pixel Pitch - Finer the pixel, less chance of Moire. Moire is unfixable in post, must replace entire screen in post if you end up with moire. 1.5-1.9mm LED pixel pitch is ideal, but it seems like people still use 2.8mm panels. If its just used for lighting you can use larger pixel pitch, even as high as 5-6mm.
Angle of View - Off axis viewing is something to consider while testing.
Frequency - The higher frequency the less strobing you’ll run into.
Bit depth - No less than 10 bit color.
Color Space Coverage - A lot of the panels are almost full rec2020, but need to know how well they cover the color space.
Brightness - They use nits for brightness, need to make sure if you are going to use them for lighting that they will be bright enough.
Reflectance - Some of them have a reflective character.
Virtual Lighting
Highlights tend to feel much hotter than you would expect, its best to dial it down.
Lighting with the Volume
You're able to create invisible soft sources above the set that are not seen on camera or on the wall.
Can also create different shapes of either light or of negative fill on the wall. The issue with this is you can have spill from the wall onto the wall itself. You can still use standard grip equipment to shade parts off the wall, so you can put a flag on the wall to shade light off of it, or you can put a flag up against the wall to act as a sider.
You're also able to reduce brightness of LED wall surrounding you to act as negative fill.
IES profiles - a map of what the light does. Most manufacturers will have IES profiles of any existing lighting instrument. So you can upload the IES profile of a light to the unreal engine to give the light seen on the set a different look.
To increase interactivity of lighting you can duplicate the video higher out of frame and increase the brightness of that duplicated video, it also doesn't have to be the exact same video, but can be nice for interactivity. Could also use additional LED panel to do the same thing which gives you more control. Could also use actual lighting fixtures as those would give you better CRI.
Baked Lighting vs. Changeable Lighting
By baking in the lighting you’ll help improve performance for the computers running the LED wall. Basically to have more options on the day, it will require more computing power to render reflections and other things that happen as a result of the change in lighting. Should be cognizant of how asking for flexibility can cause issues with computing power.
Conventional Lighting
The LED Wall Panels have 3 narrow band emitters, this is what allows them to have a wide color gamut. This allow makes their color rendition terrible. As a result we need to supplement the LED wall with our own lighting to fill out the spectrum. This does depend on what you are shooting though, more skin tones mean more problems. You are able to use x, y readings from panels to match an LED light to the wall. May still have to adjust it slightly after, but x,y will get you very close.
Panels are going to be in the 5600 range so tungsten fixtures will most likely not be used.
Use Controlled Sources - The goal is to keep all of the light off of the wall. Especially in darker scenes, need to minimize it or it will reduce contrast in the walls image.
Unreal Controlled DMX - Can patch video wall into your lighting fixtures.
Display panels don’t have good CRI.
LED wall Operator can help you with minute color correction at the base of the wall to help with blending.
Working with the LED Volume
Frustrum - Field of View of the camera.
Can also do limited color correction and changing the wall on the day. Having to rebake the entire load is very time consuming, you'll want to avoid that.
Can figure out white balance by having them put up a white card virtually on the wall and then white balancing off of the wall.
Volume operators will map the lenses to volumetric focus. So when you rack focus, the focus will be affected by the wall as well. There will be encoders that are similar to FIZ motors that are placed on the camera to tell what the iris, focus, and zoom is set.
Shallow DOF is preferable so you can throw the LED out of focus.
Give actors an additional monitor of the background so that they are able to take their cues from the background. For example if the plate turns left, they would want to turn left in the car.
The wall will only really give you 24fps, you need more computing power to get higher. Important to make sure you are at least getting 24, otherwise you will have issues.
Possible LED Wall Issues
Genlock - Everything must be gen locked in order to prevent scan lines
Shutter Sync - This is a setting within the displays which allows you to shoot off speed and cut shutters.
LED wall curvature - may see the curve of the LED wall, to avoid this need to keep light off the LED wall as much as possible
Tearing - This is where the sections of the wall are torn.
Latency - When you move the camera, the volume doesn’t move at the same rate.
Judder - If the rendering computers are not able to keep up with the enviornment you’ll see judder, need to make the environment simpler.
Color Shifts - This may happen due to overheating.
Tracking Errors - Not being able to track a camera
Asset Wiggles - CG elements in the wall will begin wiggling, which is related to computers that are not powered enough.
To solve these problems is you can turn the LED volume green, but now you have ended up with a very expensive green screen.