It only took two days to film Wentworth Miller's role