Starman - David Bowie
A music video using AI
The core objective of this project was to create a visually enchanting music video where animated lyrics synchronized harmoniously with the rhythm and emotions of the song. The primary innovation lay in leveraging advanced AI technologies to interpret the lyrics and produce captivating visual representations.
The Process
At the heart of this project is the use of AI and also the scale. If by one side it would be artistically impressive to build a video from the lyrics of a song alone, for the other it would be great to create a set of codes and apps that could streamline this workflow.
Step 1: preparing the Lyrics using Node.js
At the project's inception, I employed my programming expertise with Node.js to develop a bespoke JSON lyrics format. This format encapsulated essential components for each lyric: "lyric" (the text content), "in" (the start frame number), and "out" (the end frame number). This data structure served as the conduit through which the AI could comprehend the lyrical content and create corresponding visual sequences.
Of course, this could have been a copy/ paste job, although by using Node.js I was able to create an app that build it in seconds, saving precious time for the future projects.
Step 2: AI-Driven Expression in After Effects for Automated Animation
In After Effects, the true magic unraveled as I have artfully implemented an AI-driven expression. This sophisticated expression ingested the JSON lyrics, allowing the software to seamlessly animate each lyric based on the "in" and "out" frame numbers without keyframes.
With the JSON lyrics data deftly integrated, I embarked on a meticulous review and refinement process in After Effects. They fine-tuned the "in" and "out" frame numbers for each lyric with precision, ensuring an impeccable alignment with the song's music and emotional nuances. This diligent process culminated in avoiding using AF keyframes, saving several hours on this song only - more and more hours minding the scale on future songs.
Step 3: Engineering AI Image Generation with Creative Flair
The project's centerpiece was the ingenious engineering of AI prompts from the lyrics. Through skilful programming, each lyric line was transferred to a new file format, which was expected by Stable Diffusion: "<frame number> : <prompt>". These have also received tailored details and styles, breathing life into the visual imagery generated by the AI. This step imbued the music video with a creative flair and unique aesthetic.
From this step, I start using Python in Google Collab to implement the code, and use its resources for the AI generate the images. Each image was saved individually in a Google Drive folder, regarding the frames sequence.
Step 4: Seamlessly Interpolating with A
A second AI system was seamlessly integrated into the workflow to interpolate the keyframes generated by the first AI. This harmonious fusion of AI technologies ensured smooth transitions between frames, heightening the video's visual appeal and coherence. Below is a view in how the AI produced the images, although accelerated 300 times.
Step 5: Rendering the Spectacular Final Video
The project reached its crescendo as I employed ffmpeg to render the video inside of Google Collab still. Witnessing the culmination of AI-driven image generation and automated lyrics animation, the final result was nothing short of spectacular—a music video that captivated the senses and emotions, transcending the boundaries of conventional creativity.
Conclusion
In conclusion, this project portfolio exemplifies an extraordinary fusion of creativity, technical prowess, and AI-driven innovation. The workflow's extreme level of development and scalability ensures rapid creation of new videos from prompts, paving the way for future projects with unparalleled efficiency. Embracing Google Collab empowered seamless resource allocation across remote machines, while the ability to download the final result after video export epitomises a fully automated and future-ready approach. By eliminating manual work and embracing cutting-edge technologies, this project serves as a testament to my ability to push creative boundaries, making me a trailblazer in the realm of AI-powered video production.
