Good Shout

Things are a bit loopy when it comes to technology at the moment. We have some very useful tools popping up, but the folks evangelising them are talking about creating a machine god, data centres in space and operating at a loss of 34 billion dollars a year. To be frank these chaps are getting quite tedious, and for those of us operating in the real world – the juice is no longer worth the squeeze of dealing with them. Unfortunately there have been some actually useful applications for LLM (Large Language Models …you might have heard of them) technology, but you can’t incorporate a tool into your work flow when there are so many open questions about stability of the providers, and their future. There’s also open questions about privacy, security and the environmental impact of LLMs. So in the hope of finding a less controversial solution, I have been playing with some locally hosted LLMs.

I’ve been pleasantly surprised! The main goal was summarising a meeting transcription and pulling action points – which it managed easily, as well as summarising and taking questions about some academic writing. It all works well, and on very low spec computing equipment as well. I used a 10 year old Thinkpad X270 which worked admirably (Although did sound like a jet engine on occasion). So buying a dedicated local LLM laptop for less than £100, that you could just keep offline for maximum peace of mind, is actually quite a compelling proposition.

The process I followed was:

Install Ollama – the programme that facilitates your interactions with the model – this is the ‘middle man’ between you and the LLM

Tell Ollama to download an LLM – the model itself (I went with llama 2 but there are some other options listed here)

Then run Ollama and start the model of choice and get to work.

It is all very straightforward! Installing through Linux terminal was maybe 4 commands total, taking less than ten minutes outside of download times.

The commands were:

curl -fsSL https://ollama.com/install.sh | sh
To install Ollama
ollama pull llama2
To install the model
and
ollama run llama2
To run the model

Transcribing

he model will handle text no problem, but if you are looking to transcribe a recording as I was – I recommend using Whisper seperately. I’ve used alot of transcribing solutions over the years, and this is easily the most impressive. there’s a project that adds a GUI if you’re sick of dealing with strictly text based commands, and also a really good subtitles plugin for the free version of Davinci Resolve based on Whisper. Once you have your transcription, you can simply copy and paste it into Ollama and ask for a summary and action points for example.