# CoSketcher: Collaborative and Iterative Sketch Generation with LLMs under Linguistic and Spatial Control

## Introduction

Sketching serves as both a medium for visualizing ideas and a process for creative iteration. While early neural sketch generation methods rely on category-specific data and lack generalization and iteration capability, recent advances in Large Language Models (LLMs) have opened new possibilities for more flexible and semantically guided sketching. In this work, we present CoSketcher, a controllable and iterative sketch generation system that leverages the prior knowledge and textual reasoning abilities of LLMs to align with the creative iteration process of human sketching. CoSketcher introduces a novel XML-style sketch language that represents stroke-level information in structured format, enabling the LLM to plan and generate complex sketches under both linguistic and spatial control. The system supports visual appealing sketch construction, including skeleton-contour decomposition for volumetric shapes and layout-aware reasoning for object relationships. Through extensive evaluation, we demonstrate that our method generates expressive sketches across both in-distribution and out-of-distribution categories, while also supporting scene-level composition and controllable iteration. Our method establishes a new paradigm for controllable sketch generation using off-the-shelf LLMs, with broad implications for creative human-AI collaboration.

## Method

<img src="README.assets/method_pipeline.png" alt="method_pipeline" />

## Results

Here is a demo of sketch generated by our method.

<img src="README.assets/method_demo.png" alt="method_demo"/>

Here are some comparison for sketch generation methods.

<img src="README.assets/exp_object_ID.png" alt="exp_object_ID"/>

<img src="README.assets/exp_object_OOD.png" alt="exp_object_OOD" />

<img src="README.assets/exp_scene_res.png" alt="exp_scene_res"/>

## Requirements

### Environment

You can use the following command to install the environment.

```bash
conda create -n cosketcher python=3.10
conda activate cosketcher
pip install -r requirements.txt
```

### API Key

The system requires an Anthropic API key. If you don't have one, [create an Anthropic account](https://console.anthropic.com/login) and follow the instructions to obtain a key.

Once you have the key, save it in the `.env` file:

```.env
ANTHROPIC_API_KEY=<your_key>
```

## Start Sketching

To begin sketching, you can run the sketch iteration system by running:

```
python app.py
```

You can interact with the LLM through natural language, along with skeleton and layout handles. The rendered sketch is on the middle canvas. Stroke and layout information is also listed in the panel on the left. 

CoSketcher  support comprehensive operations including initial generation, incremental sketching, object repositioning, and fine-grained redrawing. Feel free to sketch with it!