README.md

# TinyStories: Language Model Experimentation

## Project Overview

This project aims to reproduce and extend the experiments from the paper "TinyStories: How small can language models be and still speak coherent English?" by Eldan, R., & Li, Y. (2023). Our goal is to investigate the feasibility of training a small language model with only 8M parameters to produce coherent and fluent text in a specific domain, using the GPT-Neo architecture as a reference.

## Team Members

- Collin Dang (cdang38)
- Seulchan Han (paulh27)
- Xumin Li (xumin)
- Jingwei Ma (jingweim)

## Paper Citation

Eldan, R., & Li, Y. (2023, May 24). TinyStories: How small can language models be and still speak coherent English?. arXiv.org. [https://arxiv.org/abs/2305.07759](https://arxiv.org/abs/2305.07759)

## Data Source

The dataset used in this project, TinyDataset, is accessible through Hugging Face at [https://huggingface.co/datasets/roneneldan/TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories).

The arxiv dataset snapshot used in this project is downloaded on Mar.5th 2024 and can be found here: https://drive.google.com/file/d/1achP4CYEjp1nN2UCMuTiMpO4COIhck_F/view?usp=drive_link

## Setup and Usage

### Cloning the Repository
```bash
git clone --recurse-submodules git@gitlab.cs.washington.edu:cdang38/nlp-final-project.git
cd nlp-final-project
```
This may take a while because the TinyStories submodule is massive.

### Setting Up the Virtual Environment
```bash
python -m venv venv
source venv/Scripts/activate
```

### Installing Dependencies
```bash
make install-deps
```

### Freezing Dependencies
```bash
make freeze-deps
```

### Formatting and Linting the Code
```bash
make py-fmt
make py-lint
```

### Running the Project
Ensure you have a `.env` file with the the OPENAI_API_KEY, ex.
```
OPENAI_API_KEY=XXXXXXXXXXX
```
Then, run the main script.
```bash
python main.py
```