(1) Goal
Explore the capabilities of generative neural networks and create an atmospheric image series inspired by the works of Andrew Wyeth.
Paintings by Andrew Wyatt
Wyeth’s central theme is quiet provincial life, which, however, bears no resemblance to traditional realism. Everything the artist depicts conveys a sense of unspokenness and the invisible presence of a force that drives the world—though the viewer can only guess what that force might be.
Works by Andrew Wyeth
(2) Stages
2.1
— Checking GPU availability via nvidia-smi — Installing libraries: bitsandbytes, transformers, diffusers, peft (parameter tuning, including LoRA), and the Dreambooth-LoRA training script.
2.2
— Collected a dataset of Andrew Wyeth’s paintings. I used 50 images, all sized 1:1, and uploaded them to Google Colab in a separate directory.
Preview of images uploaded to the Google Colab directory
2.3
— Using the BLIP model, a prefix, «painting in the style of WYATT, ” is added to the caption of each image title. File paths and captions are saved in metadata.jsonl.
2.4
— Utilizing the Dreambooth-LoRA technique for fine-tuning Stable Diffusion.
BASE MODELS:
stabilityai/stable-diffusion-xl-base-1.0
VAE madebyollin/sdxl-vae-fp16-fix
PRIMARY TRAINING PARAMETERS:
Resolution: 512×512 Batch size: 2 Learning rate: 1e-4 Number of steps: 500 8-bit Adam (memory optimization) FP16
— Running the training via accelerate
(3) Image generation
After completing the course, I use the DiffusionPipeline to generate images based on various prompts. Generation settings: 25 steps, FP16.
Generated images:
Prompt for the image on the left: «An old dilapidated house stands in a field, next to it are several trees» Prompt for the image on the right: «A boy and a grandfather stand on the seashore and look into the distance»
Image prompt on the left — «A girl with one dog is sitting on a hill, in the distance there is a forest» Image prompt on the right — «An old man is leaning on a large tree and looking at the bright moon»
prompt — «An old woman sits in the kitchen and thoughtfully looks out the window; outside the window is an endless field, and there is a jug on the kitchen table.»
Image prompt on the left—"portrait of a young lady, she is wearing a light beige dress, surrounded by a forest at night, the moon"— Image prompt on the right—"portrait of a young guy sitting on the grass»
Image prompt on the left — «a dark night, an empty living room, a table and four empty chairs around, stars are outside the window» Image prompt on the right — «portrait of a young girl»
(4) Generation Results
The generative model excels at reproducing the pictorial style of the artist I selected.
code notebook
https://disk.yandex.ru/d/C6BbPke-8FApzA
description of the generative model’s application
The project utilized the stable-diffusion-xl-base-1.0 model: https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0
