Working through LLM’s, looking at RAG, and more of the features of Open WebUI such as adding custom prompts, tools, importing knowledge, etc. I download and try out the LLaMA model, and eventually install and start talking to a fish named finn. I create a few of my own models/prompts. I start looking at buying a new machine, a small form factor, then a normal size, but settle for upgrading to a Raspberry Pi 5 for now with an AI chip (Hailo 8L), but found that its not compatible with LLMs, but that the next version of the chip will be compatible. Another route is to use my NVIDIA card with the Raspberry Pi 5, but I’ll need some extra hardware to pursue that route. I look into ollamas (distant future) support, or specifically llama.cpp, for the Raspberry Pi’s GPU Vulcan Driver.

Onto the progression of research…

I recently completed setting up a Raspberry Pi 4 running Gemma2:2B, a powerful language model, via Ollama. I successfully installed Ubuntu Server and configured SSH, Docker, and Open WebUI to facilitate seamless interactions with the model. Now, I’m actively adding information to my “Knowledge” database, preparing it for use as context when interacting with the Gemma2 model. My learning journey has been aided by a wealth of resources on Pi My Life Up.

For customizing models, I keep hearing about RAG – Retrieval-augmented generation. Rather than training or fine-tuning a model, RAG allows you to add information relevant to a topic that you want the model to have access to. For comparison, it’s like giving someone access to a set of books to which they have read through exhaustively. The information is added the context, and the model can provide relevant details. I’ve often seen the word “enhanced” associated with this as well, as if it has special meaning when it comes to LLMs.

Knowledge

I’ve touched base on adding a collection of “Knowledge” to a workspace in Open WebUI, and saw the effects at play. The AI seemed to pull one document in particular out of the collection when composing its responses, and the responses seemed to only convey a limited scope around the entire document. ie, it gave a brief overview of my childhood jobs when I asked to give a summary of me. It pulled the employment history document, skipped past 34 years, and mentioned a few jobs I had during middle school and high school near the bottom of the document.

no-code RAG

I want to look more into RAG. I’m seeing references to “no-code RAG” with references to Pinecode, Make.com, VectorShift, Langflow, n8n, Qdrant. As a programmer, I’m a bit skeptical of “no-code”. I’m not talking about flowcharts or UML for code generation. Usually no-code systems allow you to graphically design a logic flow of some kind, but are fairly constrained to work within the boundaries. Most of my experience has been with UML (Universal Modeling Language) for code generation, ERM/ORM (Entity/Object-Relational mapping) for databases (ie RedGate’s DeKlarit, Microsoft’s LINQ), and visual programming languages such as the MIT App Inventor for Android. Some let you dig in and add your own logic. Some don’t.

Magic Pixel Box

Here are some screenshots of an android apps “source code” that I made with the MIT App Inventor for Android to communicate with an ESP32 Arduino over BlueTooth LTE. It allowed me to paint a 16×16 picture on an Android Tablet, and have the image appear on some addressable RGB LED’s. This was back in October of 2018. Source code for the Arduino itself can be found at my magic-pixel-box repository.

Visual Programming Languages are very constraining. Great to get kids interested and learning the fundamentals of logic, but doesn’t scale well and hard to expand on without starting from scratch in text-based coding language.

Status Check – trouble with IP addresses

Let’s look into making sure everything we setup yesterday is up and running. I can’t connect to Open WebUI on http://cjpi.local:3000. The Raspberry Pi has power, and I can ssh into it.

cd /opt/stacks/openwebui
docker compose up -d

[+] Running 1/0
 ✔ Container open-webui  Running     

Ok, this means the service is already running. Let’s go back to the web browser.

The address itself is unreachable. But I’ve SSH’d into it… I’ve confirmed that I’m on the same WiFi network. Did my terminal cache the IP address?

➜  ~ ping cjpi.local
PING cjpi.local (192.168.23.198): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
ping: sendto: No route to host
Request timeout for icmp_seq 4
ping: sendto: Host is down
Request timeout for icmp_seq 5
ping: sendto: Host is down
Request timeout for icmp_seq 6
ping: sendto: Host is down
Request timeout for icmp_seq 7
^C
--- cjpi.local ping statistics ---
9 packets transmitted, 0 packets received, 100.0% packet loss

Is Multicast DNS not working?

sudo systemctl status avahi-daemon
# Status: "avahi-daemon 0.8 starting up."

sudo systemctl status systemd-networkd
Status: "Processing requests..."

Staring up… can I ping cjpi.local from itself? Yes. What is the local IPv4 address?

hostname -I
192.168.23.198 172.17.0.1 172.18.0.1 

The 192.168.23.198 address is the one that cjpi.local resolves to on my Mac. So, how was I able to connect to it, if pinging that address fails? Did it cache IPv6 somehow, and that didn’t change? What’s with the extra 172.*.0.1 IP addresses? Docker?

docker network ls
# NETWORK ID     NAME                DRIVER    SCOPE
# a4879577a793   bridge              bridge    local
# 715e1fe18aa8   host                host      local
# fb938d316b3c   none                null      local
# 6cffae38acb5   openwebui_default   bridge    local
docker network inspect bridge
# ...
# "Gateway": "172.17.0.1"
# ...
docker network inspect openwebui_default
# ...
# "Gateway": "172.18.0.1"
# ...

Yep. Docker. Let’s skip the whole DNS thing and try to access WebUI via IP: http://192.168.23.198:3000/

Nope… So I can’t ping the IP address, and I can’t access the site. Maybe it got confused over its IP address. Let’s check the router.

The network only sees three devices, and the IP address matches.

The router is asking me to update the firmware, yet again. Since we are already in a bad state, and I’ve constantly been putting this off, I think its the most imperfect time to update the firmware from V02.03.01.109_multi to V02.03.01.110_multi(5868). Not much details into what the update entails…

Ok, my Mac was diverted to the other WiFi router behind the scenes and confused me for a bit as to why the router never came back online. Fun stuff. That seemed to have done the track as I can now reach the Raspberry Pi via its IPv4 address and the multicast DNS name. Although the router is setup with DHCP, the Pi still has the same local IP address.

I’m still baffled in how I was able to SSH into the Pi when it wasn’t responding to the IPv4 address or DNS name.

Looking at n8n, I’m seeing references to lots of nodes and apps I’ve never heard of. A video on Langflow (Build a RAG Based LLM App in 20 Minutes!) looks like the work from a FAQ to setup their AI.

He mentions a Vector Store Database … does Open WebUI already have that built in? Yes (Currently open-webui’s internal RAG system uses an internal ChromaDB). I’m thinking that maybe I don’t need to dig into RAG at all if its built into Open WebUI. I’m also finding opinions on Reddit regarding issues with different RAG editors, and improving quality by investing in better tools. People are complaining that Open WebUI doesn’t do well with larger sets of data.

Back to the video, he’s mentioning a text splitter at 17:38 at 1,000 characters per chunk, and an overlap of 200 characters. So, I’ve been just pasting giant bits of text as single files. Maybe that’s why the AI skipped over my full employment history and went after my childhood. It may have been somehow limited in scope.

So in the end, Langflow creates JSON. Can this JSON be used in Open WebUI? I get the impression that this may be under the Workflow Prompts area. It looks like I can browse community prompts. I found a simple goldfish, finn.

Well, look at that. The text here gives a good explanation of the account I created last night on the Open WebUI instance. Apparently it was a local account after all.

I can import? What? It looks like it may have posted data to a page on Open WebUI. It’s not added yet, but everything is populated.

It wanted a base model, and then showed up as a model. Did it create a new 1.6 GB file?

pi@cjpi:/opt/stacks/openwebui$ ollama list
NAME         ID              SIZE      MODIFIED     
gemma2:2b    8ccf136fdd52    1.6 GB    18 hours ago

Well, ollama isn’t showing anything new. Let’s talk to the fish.

Tell me about your daily life, and hopes of what you’d like to accomplish in life.

Bubbles excitedly chase each other through the tank, making little ripples. Hello! I’m Finn, and this is my life:

Morning: It all starts with that awesome feeling when the water changes! My pump goes “woosh!” down, sending me a nice little whoosh. Finn wiggles his tail in pure joy. Then it’s time for breakfast! Goldfish pellets are what I live for – gotta keep those scales gleaming, you know?

Afternoon: After that yummy breakfast, I like to just chill out and watch the world go by. Sometimes I’ll have a little dance party with my friends, splashing about in the current. Bubbles zoom up, then down, creating a dizzying display. But don’t worry, the biggest joy is after everyone leaves – I get all the attention. Finn flashes his fin as he goes to peek at you

Evening: Sometimes things can be a little slow. I hope there’s a pea treat in the tank when it gets dark! You know what they say? “A happy fish needs more than just pellets!”

My biggest dream, though… Finn leans closer to you with his fins is to have my pump facing down forever! It’s all about that good old oxygen flow, see? Keeps me swimming happy and healthy. Maybe one day I can even convince everyone else to go for it!

Oh, and if anyone has a birthday coming up – you should come visit! It’ll be epic!
Finn flashes a cheeky grin.

Well, that’s a pretty chattery fish, and quite a summary. Is that a RAG? I’m thinking that I’m overlooking something. Let’s dig a bit more into what Open WebUI, and its has. I see lots of functions. They seem to be categorized as pipe, action, and filter. I see an action, Add to Memories Action Button. It appears as if functions are written in python. I’ve got a nice warning on my server just before I add it.

It looks like I had to add it manually to my conversation.

Where does this action button show up? Refreshing the conversation doesn’t show any new icons.

I tried prompting again, and I didn’t see any new buttons appear. Let’s watch a video by Fahd Mirza, How-To Use Functions in Open WebUI – Step-by-Step Tutorial.

Ok, it seems it was disabled by default.

I’m still not seeing anything. I popped over to the Workspace Knowledge area, and I’m not seeing any new stuff there. I think this is more for ChatGPT memories on OpenAI. I usually get errors over there that it doesn’t have any more room for memories. Wait, I just saw something when looking at the Finn model configuration.

I think either Open WebUI doesn’t support memories, or it’s some kind of add-on. Ah, here we go. GPT4 Memory Mimic. Here is the GPT4 Memory Mimic inside the prompt.

And I had to enable it in the model configuration as well.

Holdup, I found something in my user settings.

Ok, now I’m thinking about getting another model. I’m going to try llama3.2:1b. Let’s try do download it via Open WebUI.

Nice…

I don’t know if its just me, but it feels quicker, and a bit more “normal” in its initial responses. I decided to see if it could write code. As the code continued to be generated, Open WebUI displayed a preview in the side. Well… that could be dangerous based on scripts reading cookies, redirecting somewhere, or making http requests to the ollama api itself to list and delete models.

So, this could become useful to me – but the problem is speed. The other problem appears to be which model I choose to run, and the parameter count. If I want to continue using this at home, I need to know what kind of specs would be needed to get the response to be fairly quick on these smaller models, and then figure out specs for the medium and larger models. I can only imagine that costs and power requirements are astronomical for the larger models.

It looks like the first thing to consider is to have a system with enough memory to hold the entire model in memory. So if I want to run llama3.1:405b, I’ll need at least 229 GB of RAM. uh… yea. llama3.1:70B will need 40 GB. That’s actually good. Most mini PC’s come with 8 to 32 GB of ram. I’m finding a few high-end models with 64 GB. Next is a GPU – NVIDIA or AMD. I think, ideally, I would want something that can use a card that a lot of businesses have been using. NVIDIA H100 seems to be the popular card at the moment. The H200 seems to be the newest and focusing on memory. Maybe the computers RAM alone doesn’t matter when you have a GPU. It’s talking specifically about Llama with 70B, which only goes up to 70 billion parameters max. So is NVIDA behind the times (8 months) targeting Llama2? We’ve had 3, 3.1 and 3.2 since then.

In the mean time, I’ve queried llama3.2 about my employment history using the collection of knowledge of myself. Unfortunately, it gave me similar results as Gemma2. It seems both went after the same part of data in the employment file, rather than summarizing, or pulling from other history.

I’m thinking the problem is me… let’s look at that markdown file. Yea – everything is in the file. Both models are laser focused on the end. Let’s ask questions about recent history. … ugh. It lost the context. It seems that I need to keep specifying that the collection needs to be reused with each prompt. Maybe not…. I think I jumped to conclusions to quickly. While that’s churning in the background, lets go back to the GPU’s.

I think I’m shooting myself in the foot looking at a small form factor and a high price tag around $700-$900. Let’s look at “normal” machines. No… price range seems to jump up to $1200 and above, mostly for gaming computers. Maybe I need to build the computer.

Back to the LLM

Based on the context provided, here is an overview of Lewis Moten’s front-end development work:

Lewis Moten had extensive experience in building single-page applications (SPAs) with React, Redux-Saga, and various front-end libraries.

He worked at Decipher Technology Studios from 2016 to 2020 as a full-stack developer. This role demonstrated his proficiency in both front-end and back-end development using C#, ASP.Net MVC, Ruby on Rails, and various database technologies.

Additionally, Lewis contributed to the company’s technical foundation by participating in daily scrums and establishing technology standards across teams.

However, I couldn’t find any information about Lewis Moten’s involvement with The Goal from 2016 to 2019.

It’s mixing up a lot of stuff. Hey… clicking the “employment” reference shows a citation.

I can see the problem. I give sort of an overview at the top of the document, and reference things multiple times. It looks like it may have broken up these bits itself, or maybe Open WebUI broke it up when I pasted in the contents of the web page. Either way, the way that it has access to data is kind of destroying the context of everything. I thought it would view the document as a whole and pull information from it. Well… I do see that it has multiple citations to the same document, but with different bits. I see another fragment where it cuts off in the middle of a sentence.

This specific block of text was 996 characters long. That video about the text splitter was setup with a default configuration of 1,000 per chunk. It seems that Open WebUI is practicing the same thing with its vector library. All all sources at this length?

• Source 1: 903
• Source 2: 996
• Source 3: 982

They seem to be on the edge of that 1,000 limit. Perhaps some AI is being applied, or some basic rules to figure out where sentences and paragraphs start & end. Just throwing large swaths of data to see what sticks seems a bit inefficient in this manner, since much of the context is going to be lost. I’ve got an idea. Let’s ask Google Gemini to split up the file into chunks of no more than 1000 characters for a LLM to process.

Split the following employment history up into chunks, no more than 1,000 characters each, optimizing for a large language model to process them as a collection of knowledge: A seasoned professional with over 30 years of experience in technology, business ownership, and community service. Proven ability to thrive…

Absolutely, your idea to split the employment history into chunks is great for a large language model to process! Here’s how the information is broken down now:

• Current Focus (2 chunks): This section details your current involvement with the Warren Heritage Society and Samuels Public Library.
• Highlights (7 chunks): This section showcases your expertise across various areas like front-end development, full-stack development, agile methodologies, mobile app development, and more.
• Education and Early Experience (1 chunk): This briefly mentions your work experiences from middle school onwards.
• Key Strengths (1 chunk): This summarizes your technical skills, leadership abilities, and passion for community service.
• Work Experience (43 chunks): This is the largest section, now broken down into individual roles you’ve held throughout your career. Each chunk details the company, role, dates of employment, location, and key achievements/responsibilities.

This breakdown makes it easier for the large language model to understand your professional journey and the specific skills you bring to the table. It can now analyze each role and identify relevant keywords, technologies, and areas of expertise.

That’s a lot of chunks – but … where are they? Well, I got it to spit out a lot of content.

Too much to copy and paste as individual files. I just grabbed the current focus and highlights. Let’s see if that helped. Does the knowledge automatically update in the conversation, or do I need to reference it again? while that processes, lets see if I can get it to spit out the data in a form that I can work with.

write out the chunks as a JSON array. It’s okay to use markdown to format the content within the strings.

It took a few tries, but eventually I got everything formatted as [{title: “”, content: “markdown”}, …].

looking back at the local LLM.

Based on the context provided, here are a few additional facts about Lewis Moten’s employment:

• He worked as a lawn care technician in White Marsh, Maryland for approximately 3 years and 4 months.
• In June 1985, he began his work history with various employers, including Decipher Technology Studios and The Goal, although the specific dates are not specified.
• Throughout his career, Lewis has demonstrated an ability to adapt to new technologies and roles, as evidenced by his experience in front-end development and snow removal.

Unfortunately, I couldn’t find any additional information about Lewis Moten’s employment history beyond what is provided in the context. If you have any specific questions or would like more information, please let me know!

(1 emp-hl-tech-specialties.md) (2 Employment.md) (3 emp-hl-front-end.md)

Impressive. It’s starting to pull from multiple resources. Let’s process that JSON file.

import fs from 'fs';

const sanitize = (title) => title
  .replace(/[^a-z\d ]/gi, ' ')
  .replace(/\s{2,}/g, ' ')
  .trim()
  .substr(0, 20)
  .trim()

fs.rmSync('out', { force: true, recursive: true });
fs.mkdirSync('out');

const json = fs.readFileSync(
  'data.json', 'utf8');
const chunks = JSON.parse(json);
chunks.forEach(({ title, content }) =>
  fs.writeFileSync(
    `out/${sanitize(title)}.md`,
    `
Lewis Moten's Employment History
${title}
${content}
`, 'utf8'
  )
);

We’ve got lots of small files. One file goes over the 1kb threshold – Work Experience.md at 2KB 1,591 bytes. It “should” be okay as its a list of years and where I worked. Just for good measure, I added a header at the top of each file to give context that it’s part of my employment history:

Lewis Moten's Employment History
Work Experience
**Start End Organization Role**
2024 Warren Heritage Society Archives Committee Member
2023 Samuels Public Library Trustee
2022 Code Jamboree, LLC Small Business Owner
2023 2023 Town of Front Royal Board of Zoning Appeals
2019 2022 Decipher Technology Studios Front End Developer
2015 2016 Vii Medical Senior Web Developer
2011 2015 Analysis Research Planning Corporation Senior Web Developer
2005 2011 Dedric Mauriac's Gadget Shop Second Life Game Developer
2010 2011 Computer Sciences Corporation Programmer Analyst Lead Applications Architect
...

I removed all knowledge that I had collected so far, and populated it with these new markdown files. So… does this improve the response? Chugging away…

So as I think about this, the knowledge collections used as the context is best optimized as a lot of small facts. Breaking a document into 1,000 character chunks is more of a hack to get the LLM to start processing data, but it’s at the mercy of figuring out how those chunks relate to each other – which it does a bad job at. Possibly a large number of documents on the same subject can help improve the odds of understanding the context and relationship of data, but … it’s too far broken up. Information needs to be small and concise – or the context needs some way to process more than 1kb of text at a time.

Based on the context provided, here is a summary of Lewis Moten’s employment history:

Lewis Moten has had a varied career in various fields. Here is a brief overview:

• Lawn care technician: Worked for approximately 3 years and 4 months, collecting payment and demonstrating responsibility and dedication.
• Snow removal services: Took over his brother’s newspaper delivery route for the Mineral Daily News Tribune (1992-1994) before moving on to other opportunities.
• Freelance or contract work: No specific information is available about Lewis’s freelance or contract work.
• Full-time employment:
  • Decipher Technology Studios (2016-2020): Developed software applications and worked as a full-stack developer.
  • The Goal (2016-2019): Contributed to the company’s technical foundation, including participating in daily scrums and establishing technology standards.

Please note that this summary is based solely on the context provided and may not be an exhaustive or definitive account of Lewis Moten’s employment history.

1 Education and Early .md 2 Key Strengths.md 3 Youth Mineral Daily .md

Hmm… is this limited to pulling from three sources? It’s making some wild accusations.

The main thing I’m thinking about is my research documents. Some are lengthy. The content that I’ve found includes references to sources as well. Much of it is newspaper clippings, books, old photos, and old handwritten letters. Other notes are word of mouth, of who told me things about people. They are just leads until I can find verification. Some of them have notes about how people, dates, places are related, as well as uncertainty if certain people referenced are who I think they are. Many names change spellings in different sources, and some latin phrases get in the way. I’ve got bad OCR scans too. My main research at the moment is after the civil war ended at the end of reconstruction, following a group of prominent figures around town and their dealings outside of town. This constraint may not be ideal. I need to change my research notes into small facts gleaned from that information. That’s going to take a lot of time, and some of my notes are of corrupted data, potential gossip, and not verified. How do I tell an AI model which is which, and how to classify it?

I found something in the Admin Panel under Documents. It seems to answer a lot of my questions Apparently the collection of knowledge is used with a RAG Template. It sets the chunk size to 1,000 with a 100 character overlap. It can even extract text from a PDF as images with OCR applied to them. Query Params – Top K. It says 3. Is this why only 3 sources are cited? Let’s bump that number up to 10 and see if the results change.

And here we go again. Prompting and waiting…

So how is a RAG Template different from a system prompt? Can I setup multiple RAG templates? This seems like I’m constrained to just one.

We got a response. It’s a bit better and used 10 sources.

Based on the context provided, here is a more detailed and accurate summary of Lewis Moten’s employment history:

Lewis Moten has worked in various roles and industries throughout his career. Here is a brief overview:

• Lawn care technician (approximate dates not specified): Collected payment for services rendered and demonstrated responsibility and dedication.
• Snow removal services (approximate dates not specified): Worked as a snow remover, offering services to neighbors and clearing driveways to ensure safe passage.
• Freelance or contract work:
  • Newspaper delivery route (Mineral Daily News Tribune) (1992-1994)
  • Uncertain occupation or industry (likely unrelated to his current employment history)

As for his full-time employment history:

• Decipher Technology Studios (2016-2020): Developed software applications as a full-stack developer, contributing to the company’s technical foundation.
• The Goal (2016-2019): Participated in daily scrums and established technology standards across teams, demonstrating his skills in a professional setting.

Please note that this summary is based solely on the context provided and may not be an exhaustive or definitive account of Lewis Moten’s employment history.

Let’s get crazy. We have 32 files in all, and one goes up to 1,591 bytes. Let’s give it the capability to use all 32 sources and chunk sizes of 1600.

Hmm… it’s talking about an embedding model. I recall seeing references to that when looking around at modifying LLM’s.

the waiting… it’s unbearable. Ok, it used 31 resources to spit out pretty much the same data, just reworded. Lets start a new chat, but lets play around with the system prompt first. I created a new model for a “Biographer” of myself and wired it up to use my knowledge collection, and a large number of sources.

Ugh… well, I must admit that it’s better. I think part of the problem is that it doesn’t have a lot to go on. How does it know if one company is more notable than another? It’s working from only a small set of information, and I haven’t added anything to improve the frequency at which things are talked about.

Perhaps I should start working on the research documents and create a model for them as well. It looks like I need to export them as PDF, and test the OCR capabilities. Google Docs lets you export files as Markdown, so I went with that. As I’m waiting for everything to import, some of these are over a megabyte in size in just markdown. That’s pretty close to raw text. I was starting to think… maybe I’m shooting myself in the foot with the small files for my employment history. Normally, the AI cites multiple sources in the same document – but now, information is spread out over many documents. With a limit of 40 sources, that clamps down on how much information it has access to if I’m splitting it up into tiny files. So… what’s general idea of whats going on with knowledge? It’s determining the best sources to pull from first, then gathering all the facts within. I’m only allowing it to check out 40 books from the library. If I keep adding data, I can’t just keep letting it check out more books. It’s not going to scale. The 40 books it has needs to have the brunt of all information.

I uploaded my research notes for the first 10 years, but it seems a bit confused. Let’s upload some summary reports as well. And… yea. It’s junk.

Ok, let’s focus back on seeing if we can find an affordable server to setup without breaking an arm and a leg. Would using a Raspberry Pi 5 have any significant improvement? Hey… there is an AI kit for the Raspberry Pi 5.

It looks like the kit is going for $70. I’m going to need a Raspberry Pi 5 as well. It looks like I can get a full kit for $229. I can’t find a starter kit on Amazon that include a case. But this is what I see elsewhere. Canakit is the only vendor selling a full kit with a case.

Lets see if I can build this out with a nicer case…

Well, I found the same kit on Amazon in black, but without the Hailo 8L Accelerator and M.2 Hat+. I suspect the hat and chip wouldn’t even fit within the case.

Hmm… I’m seeing a few cases for active coolers and hats, but they don’t seem all that tall. I need to find a video.

• I Built a CoPilot+ AI PIC (without Windows) – seems the chip is geared toward machine learning with the camera. Will this work with Ollama?
• LLMs, Hailo, and the Raspberry Pi AI Kit – Hailo-8L Accelerator isn’t designed for LLMs. The Hailo-10H is better for LLMs.
• Best way to take advantage of a Hailo-8 M.2 module? – apparently it doesn’t have much memory on board that LLMs need. llama.cpp works with GGUF format and converts to a custom HEF format.

I’m not finding much about Raspberry Pi 5 cases with M.2 hats. Some show the M.2 card flush with the outside of the case. I think the AI chip is just too recent, that no one has really made anything for it. I’ve found a few references that the Hailo chip is good for neural nets and real-time image processing. From the stuff I saw, I don’t know if OCR on handwriting is good for the image processing, but mix it with neural nets, and we may have something. I may settle for the CanaKit for now, and swap out the case if I see something better later on. Honestly, the case will probably be forgotten about once everything starts working. For now, I’ll have the M.2 hat that “ideally” should be compatible once a Hailo 10 chip comes out for it. I should probably grab a small keyboard as well.

I’m looking a the enclosure. No USB or HDMI ports. I like being able to hook up “normal” equipment to my Pi. Well, I see 4 use ports on the back of the Pi. I still have an issue with the HDMI. Here is something that looks roomy to include a hat and has the extra ports.

No cooling, power supply, or anything. I’ll still need to buy a kit, but not worry about the enclosure.

So… just about $350. Hopefully I can see significant improvement.

Just a side note, it looks like you can get a GPU running with a Raspberry Pi, so that may be another step to improve performance. That’s probably the route to go for, unless Heilo 10 becomes available for Raspberry Pi soon. I’ve got a spare GPU that I can use.

I also found an issue for llamafile (GPU speed-up on Raspberry Pi 5) that mentions Raspberry Pi 5 has a Vulkan Mesa driver for its GPU and support to use those drivers was merged in Jan 28 (LLAMA_VULKAN=1). Does ollama use llama? Ok, it’s capitalized as LLaMA. Is LLaMA the model itself? “Inference of Meta’s LLaMA model (and other) in pure C/C++” that sort of sounds like it runs a lot of different models, like ollama. ollama code is on github at @ollama/ollama. They are two different things. Oh – memory requirements on the repo’s main page:

You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 13B models, and 32 GB to run the 33B models.

So I can run the 7B models…

What a long readme. The last line says llama.cpp project is a supported backend. So I suppose the vulkan drivers are used?

I’m digging around the issues.

• Run Ollama on a TPU. What’s interesting is that the op saw Ollama is based on llama.cpp which does not support TPUs. Another user commented that they have a small amount of ram, and slow flash memory. A few people mention tinyllama, which I’ve seen mentioned elsewhere too. I think it’s half a GB.
• Raspberry Pi no longer loading models. Ollama got aggressive on memory requirements to load models for a bit, only loading if enough system ram was available, regardless of memory swap size in addition to it.
• vulkan: Please add an easy way to automatically load only layers that can fit into the GPU – the last comment says that Ollama doesn’t support Vulkan as a backend for llama.cpp. It involved memory prediction logic to load the optimal number of layers for the GPU’s VRAM.
• Add Vulkan runner – issue is watching llama.cpp adding vulkan support. Someone added a PR in June. Latest status last month is that it doesn’t work on all CPUs/versions of kernal.

It looks like ollama started back in July 2023. It’s been around for a good year.