The first step is to upload a video to the app. The video is then transformed into a series of static images and sent to GPT-4 Turbo with Vision for transcription.

This is because, currently, GPT-4 Turbo cannot transcribe videos directly, so we have to break it down into a series of images first, like a slideshow verison of the video.

The idea is to give GPT-4 enough context, along with any user settings like style and persona, to understand the video and generate detailed a transcription.

Here we are using a stock video of a penalty shootout to generate some VoiceOver commentary for it. The twist being that we can choose from a variety of voices and styles.

We can choose the style to be a poem, and the persona to be of David Attinborough. The result is what you see here in the gif, a decent attempt at a poetic commentary of the penalty shootout in the video (full video below).

The commentary is impressively accurate to what is going on in the video - inferring that it's of an indoor penalty shootout for starters - and takes into account the sequence of events, like the striker stepping forward and the keeper subsequently crouching. All these details are picked up and written into the poem.

This is a testament to GPT-4's inference capabilities, and why it is so powerful for many use cases, generating VoiceOver being one of them.

Of course you don't have to make GPT follow a particular style or persona, you could fine-tune the model however you'd like to get the results you want. The possibilities are endless.