README

diff --git a/README.md b/README.md
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0c9bc4f1ae9b781111a8b1b2fdcd74abe195692f 100644 (file)
--- a/README.md
+++ b/README.md
@@ -0,0 +1,92 @@
+*This project has been created as part of the 42 curriculum by agilliar.*
+
+# Description
+
+Retrieval Augmented Generation (RAG) is a method of language generation meant to answer questions after having retrieved related sources.
+This CLI enables indexing of a corpus of document then doing retrieval and generation on it.
+
+# Instructions
+
+To run the CLI, simply enter the workspace with:
+```sh
+uv sync
+```
+Then run it using:
+```sh
+uv run rag --help # Dislays the available commands
+uv run rag index --help # Dislays arguments for a given command
+```
+Or after entering the venv
+```sh
+rag --help # Dislays the available commands
+rag index --help # Dislays arguments for a given command
+```
+
+# Resouces
+
+- [Transformers documentation](https://huggingface.co/docs/transformers/index) used for generation (specifically Qwen/Qwen3-0.6B).
+- [BM25 wikipedia article](https://en.wikipedia.org/wiki/Okapi_BM25) algorithm used for source retrieval.
+- [Python fire](https://github.com/google/python-fire) used for CLI handling,
+- [Pydantic](https://pydantic.dev/docs/validation/latest/get-started/) used for validated inputs and outputs in json.
+
+*No LLMs were used in the making of this project.*
+
+# Unimportant stuff I was required to include
+
+## System architecture
+
+There isn't much to describe here, each step in the pipeline feeds into the next.
+The full pipeline is:
+```sh
+# Every command has configurable paths but defaults are somtimes provided
+rag index path/to/corpus
+rag search_dataset path/to/dataset path/to/search/results/dir
+rag answer_dataset path/to/search/results path/to/answer/dir
+# Checking search results:
+rag evaluate path/to/search/results path/to/expected/results
+```
+
+## Chunking strategy:
+
+Input files are chunked differently depending on file type.
+Each line is assigned a depth depending heuristics:
+- For python: Indentation
+- For text: `#` header level, then empty lines, then lines with text
+After that, while a chunk is too big, split on lines with increasing depths level into subchunks.
+On the way back, chunks that are small enough are merged back to size.
+As a fallback, single line chunks are split on words then at chunk size with no regard for structure.
+
+## Retrieval method
+
+There is nothing much to say here, it is a dead simple BM25 using simple bags of words to represent a document, tracking a few useful numbers to not recompute them every time.
+The bag of words are constructed by ignoring non-alphanumeric characters, normalizing to lowercase, and ignoring stopwords from a dataset.
+
+## Performance analysis
+
+The retrieval and indexing are adequately fast for python, at about 80 queries/s on my system, and only a few seconds taken to index.
+Recall@k (correctly recall a source in k samples) is above the expected values. No tuning was done except the aformentioned stopword elimination and normalizing.
+Text generation is inhenrently slow and especially so here, especially as a CPU workload. For an unknown-to-me reason, transformers doesn't immediately start streaming output when reaching a query. This was not investigated further than a simple search which brought up no obvious "I did something silly" answers.
+Since transformers and associated are incredibly heavy to load, they are only imported when required to avoid paying the cost for unrelated tasks.
+
+## Design decisions
+
+The general structure was imposed, though it is somewhat sensible.
+Implementation were generally made with the naive but effective approach, and big performance gains were obtained when they were straight forward.
+The pydantic models given were overengineered, and overly verbose. Since they are required they were put in a dummy module, but simpler (ish) models were used instead. There were some idiocies as the ground truth datasets did not match the expected output, making things messier.
+
+## Challenges faced
+
+Some time was wasted trying to utilize the pipeline generation and have the LLM output its EOS token to finish, which was unsuccessful.
+Other than that, making sense of the given pydantic models was one of the bigger "challenges".
+Implementing BM25 was one of the more interesting parts, everything else was just finding the right docs to plug things into each other correctly.
+
+# Extra notes
+
+The required tooling (flake8, mypy) is remarkably slow and clunky, and so most of the development was made using the astral stack:
+- uv
+- ruff
+- ty
+
+Python fire completely ignores type annotations for conversions, leading to bad behavior in certain cases.
+
+Please no more AI projects :)