Running your own local AI.

Grimlakin

FPS Enthusiast
Joined
Jun 24, 2019
Messages
7,394
Points
113
I did this in windows largely following the instructions I found here:
https://www.howtogeek.com/881317/how-to-run-a-chatgpt-like-ai-on-your-own-pc/

For the last bit when it has you use a command line to download the actual AI engine... or whatever you want to call it. It fails. If you go into Docker expand the project you can launch a web interface and download the AI engine of your choice.

Be warned... when they say this takes a LOT of CPU and a LOT of Ram they are not effing about.

I run the mid tier 13b engine and it easily can max out my 5900x if I let it have the threads. That and the Memory load pretty much eats all it can without pushing the OS over.

It was fun to setup and now know I have a built in integrated Linux VM in my windows OS.

To be fair I'm running windows 11 pro. I would suggest the same for following this path.

Now I am looking at going to 64 gigs of ram by adding another 32 gig kit to the one I have already. Price isn't bad either today.
 
What exactly can you do with this AI? I mean, is it basically like running your own ChatGPT with an interface where you can interact with it, ask it questions, etc...

I got a 128 core EPYC with 256GB of RAM I might play with over the summer while it's idling. lol

Does it only use CPU or does it have GPU capabilities also? The rig has 5x P100s in it. Alternatively, I could throw it in my 64 core EPYC setup with 6x P100s or my 48 core EPYC with 6x 3070s. I wonder which would be better suited for the task assuming it can utilize the GPUs?

edit
Seems it can't utilize the GPU. So the 256 thread EPYC is what I will install this on and play, but I intend to try out the bigger version aka 30b.
Seems their is an AI that can utilize the GPU, but it seems to be based on the 7b version which according to what I read is a lot dumber than the 13b and 30b versions.
 
Last edited:
Yea this one is pretty simple by comparison. I was surprised by the flexibly of the mid tier version. Would love to know your impressions of the max tier version.

What I'm trying to find now is a AI you can point to a site base URL to train.

Because if I can build a porn chat bot trained off of literotica.com to make money with... I'll be a millionaire in months.. lol.
 
I wonder if you could connect it to the Internet and tell it to "learn everything" and it learn everything it doesn't already know from the Interwebs. Definitely not dangerous, right?
 
LeRoy_Blanchard said:
I wonder if you could connect it to the Internet and tell it to "learn everything" and it learn everything it doesn't already know from the Interwebs. Definitely not dangerous, right?
Click to expand...
That is part of the whole responsible use. An ai as we have them today can ingest information but it has no capability to jusdge if a piece of information is correct. That's why you need to have the ability to see the source of information used and also have a subset of information sources not just everything everywhere.

The problem here is that repetition of inaccurate information will make it seem more accurate to an ai scrubber.

Would you trust an msnbc data source over a bbcnews data source? Things like that are going to be issues to hurdle.

The best way to use an AI today is as a limited subset data aggregator and query engine. Letting it grow as you or a AI appointed board approves data sources it is limited to for it's use case.

At least that's how I see it.
 
Grimlakin said:
The best way to use an AI today is as a limited subset data aggregator and query engine. Letting it grow as you or a AI appointed board approves data sources it is limited to for it's use case.
Click to expand...
It takes a village to raise a baby approach eh?
 
Peter_Brosdahl said:
It takes a village to raise a baby approach eh?
Click to expand...
That's actually very true.

Imagine an ai trained on a support desks knowledge base for all levels. Then s customer calls in agent takes the call. As the call is worked the ai is prompting with results and learning from the agen what results are best. Not to replace the agent but to expedite the call helpi to the scent in an automated manner.

Stuff like that is where the value add is. For normal customer service issues it could be all AI driven. System knows the calling number and associated metadata. Speaks on a manner the metadata suggests is known and comfortable to the called and is even enabled for culture based small tall around the callers interests. Ultimately govi of the caller a positive experience and representing the company well.

We're a surprisingly short period of time away from those abilities.
 
You must log in or register to reply here.
Become a Patron!
Back
Top