Monday back in Culver

LLM: Using GPT-4 to query a site

Here’s a quick incantation to use GPT-4 to ask questions about a site.

First off, you need a Python environment with these dependencies installed (assuming an environment is set up using poetry):

poetry add langchain beautifulsoup4 nest_asyncio ipython
poetry run ipython

Using langchain, load all pages referenced in this site’s sitemap.xml into a vector index.

import os
from langchain.document_loaders.sitemap import SitemapLoader
from langchain.indexes import VectorstoreIndexCreator

os.environ["OPENAI_API_KEY"] = "..."

sitemap = "https://jtmoulia.srht.site/sitemap.xml"
sitemap_loader = SitemapLoader(web_path=sitemap)
index = VectorstoreIndexCreator().from_loaders([sitemap_loader])

# And now the site's vector index can be queried
index.query("When was I last in LA?")

Note: this will cost ~0.14c using OpenAI’s GPT-4 interface. The index can be persisted.

Exercise

Leg lifts using the squat rack – already feeling tight, so these legs are going to be curled tomorrow eve.