VQGAN + CLIP pt. 2

Published: 10/15/2021

Yesterday I did my first dive into the practical usage of VQGAN+CLIP. I've spent a few months exploring STYLEGAN models, and have even trained a few of my own on RunwayML. They've been fun, and will definitly be sticking around as a fun curiosity for awhile, but there is something about VQGAN+CLIP that is extra sticky. Generating images and video from text is just cool.

I ultimately ugraded to Google Colab+ for access to faster GPUs. I'll be canceling my membership after my curiosity has been sated. They really were not kidding about the faster GPUs—the ~5 hours of processing for my first run through was cut down to ~1 hours after upgrading. More time for projects!

So, with that, I re-ran the model using almost all of the same settings. There was only one difference—instead of using the S-FLCKR model which is comprised of photographs, I used the WikiArt 16384 model which is build off of paintings from 195 different artists.

Far over the misty mountains cold || To dungeons deep and caverns old || We must away ere break of day || To seek the pale enchanted gold. || The dwarves of yore made mighty spells, || While hammers fell like ringing bells || In places deep, where dark things sleep, || In hollow halls beneath the fells. || For ancient king and elvish lord || There many a gleaming golden hoard || They shaped and wrought, and light they caught || To hide in gems on hilt of sword. || On silver necklaces they strung || The flowering stars, on crowns they hung || The dragon-fire, in twisted wire || They meshed the light of moon and sun. || Far over the misty mountains cold || To dungeons deep and caverns old || We must away, ere break of day, || To claim our long-forgotten gold. || Goblets they carved there for themselves || And harps of gold; where no man delves || There lay they long, and many a song || Was sung unheard by men or elves. || The pines were roaring on the height, || The winds were moaning in the night. || The fire was red, it flaming spread; || The trees like torches blazed with light. || The bells were ringing in the dale || And men looked up with faces pale; || The dragons ire more fierce than fire || Laid low their towers and houses frail. || The mountain smoked beneath the moon; || The dwarves, they heard the tramp of doom. || They fled their hall to dying fall || Beneath his feet, beneath the moon. || Far over the misty mountains grim || To dungeons deep and caverns dim || We must away, ere break of day, || To win our harps and gold from him!


Initial Output


Far over the misty mountains cold


loading...


The Second Prompt


To dungeons deep and caverns old


loading...


The Third Prompt


We must away ere break of day


loading...


The Fourth Prompt


To seek the pale enchanted gold


loading...


The Result


loading...