I’d like to use a local LLM for Github Copilot-esque completions. I like Copilot; it’s downright uncanny when it serves that perfect completion. However, it comes with a few drawbacks:

• Cost: $10/mo adds up, especially when I already pay for general purpose LLMs
• Privacy: even if Github sticks to its privacy policy and doesn’t keep queries for future training, sending code to third parties introduces a risk surface many organizations find unacceptable.

A local LLM sidesteps these issues

Intro

This post collects initial benchmarks of Ollama running LLM inference across on my server and my laptop: the server armed with a Radeon 6900 XT GPU and the laptop using CPU-only processing. Both setups run Arch Linux, and ROCm provides AMD GPU acceleration.

The benchmark focuses on token generation speeds (tokens/s) for various models.

The Setup

• The Server (GPU):
  • Radeon RX 6900 XT
  • 16GB GDDR6 RAM (~448 GB/s)
• The Laptop (CPU):
  • 11th Gen Intel i7-1185G7 @ 3.00GHz
  • 32GB DDR4 RAM (~26 GB/s)
• OS & Setup:
  • Arch Linux with ROCm for GPU acceleration (see archwiki)
  • ollama v0.4.2

Benchmark Results

There was a 35% - 110% speedup moving from the Intel i7 CPU to the Radeon GPU, with greater gains generally coming from the larger models (qwen2.5-coder:7b being the exception).

Python has yet another packaging tool, hatch. Here’s how to use it with Emacs.

Create a new project by calling:

hatch new PROJECT_NAME

Testing

Hatch configures new projects to use the pytest runner by default. This runner can be integrated with Emacs using the excellent python-pytest package and setting the python-pytest-executable in .dir-locals.el:

(python-mode . ((python-pytest-executable . "hatch run pytest")))

IPython

To use IPython as your project’s shell first add it to the project’s dependencies (good practice to keep it pinned to a version range):

My Emacs config is now literate; here’s the core.

Variable Definitions

;; TODO: use an actual variable
(setq-default my-sync-dir (expand-file-name "~/Sync"))

Package Management

straight handles package management.

Bootstrap straight

;;;;;
;; STRAIGHT
;; bootstrap the pkg manager
(defvar bootstrap-version)
(let ((bootstrap-file
       (expand-file-name
        "straight/repos/straight.el/bootstrap.el"
        (or (bound-and-true-p straight-base-dir)
            user-emacs-directory)))
      (bootstrap-version 7))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
        (url-retrieve-synchronously
         "https://raw.githubusercontent.com/radian-software/straight.el/develop/install.el"
         'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp)))
  (load bootstrap-file nil 'nomessage))

Configure straight

;; Fetch use-package from straight
(straight-use-package 'use-package)
;; Configure use-package to install pkgs using straight
(setq straight-use-package-by-default t)

Utility Packages

(use-package a)
;; file manipulation
(use-package f)
;; string manipulation
(use-package s)
;; other helpers
(use-package dash)

Editor Config

;; hide warnings, only let errors through
(setq warning-minimum-level :error)

;; auto-close delimiters
(electric-pair-mode 1)

;; no menu bar
(when (fboundp 'menu-bar-mode)
  (menu-bar-mode -1))
;; and no tool bar
(when (fboundp 'tool-bar-mode)
  (tool-bar-mode -1))

Backup Files

Backup and temporary files are stored in $XDG_CACHE_HOME/emacs/backups if the env variable $XDG_CACHE_HOME is defined, else they’re stored in ~/.cache/emacs/backups.

A change of plans left me with extra time in my hands today, so of course I spent it on an Emacs reconfigure. Yup, I too felt like it was a waste of time. I swear I tried to avoid it, but Python & LSP & eglot were fighting and without Python support it’s real tough to justify Emacs. I truly tried, but with just the complexity of Doom I wasn’t able to debug it within an hour.