I’m a big fan of local AI and I think it has to be the future.
It’s ridiculous l, like Bonsia AI’s Q1 models are like 3.5GB easily doing basic tasks that most people are asking 600GB flagship models.
Who on earth would pay for something that needs a basically terabyte or RAM that only performs 10% better
The industry keeps benchmarking against other labs instead of against user needs: If a 3.5GB model answers 95% of everyday questions well enough, the remaining few percent has to justify hundreds of gigabytes of weights, huge energy bills and constant cloud costs.
It’ll be “local AI” when they let me point to my own self hosted inference servers, rather than simply OpenAI, Anthropic, or Gemini as providers. Soon to include Apple provider, I guess.
I’ve used the Apple Intelligence ecosystem. The models are slightly acceptable in extremely small context windows, but they completely shit the bed for any kind of practical ad-hoc use. Even if you try to dumb it down to like 6 words. It’s trash. It couldn’t even find a picture of my finger with the keyword “finger.” It couldn’t explain basic details of my phone… it was like interacting with a shittier version of ChatGPTs first release — much shittier.
It told me I have an iPhone 18. I have a 17 Max Pro, the 18 hasn’t been released yet. I don’t plan to ever buy the 18, given the hardware regression with their charging port. But… as I said, it’s not even released yet.
That’s fine with me, actually. If that’s all my phone can handle then so be it. But, when I eventually and obviously will want something more practical, my only options shouldn’t be to pay frontier cloud models if I want deep integration with my phone. The only alternative shouldn’t be to subscribe to a higher tier iCloud+.
I can put a vpn on my phone to access vLLM or ollama locally. Why won’t they let me use that?
On android you can just use something like jegly Box and just run Gemma 4 or some other model. Decent results for an offline ai.
The interesting part is not whether Apple wins the biggest model race, but whether it changes the economics: If enough AI runs locally, every token avoided is cloud capacity nobody has to build. That is a very different business model from selling ever more cloud compute.
I’m pretty skeptical local models can hold a candle to the cloud-based ones, particularly the ones Apple trains.
Raw capability is only one metric: A local model probably will not beat the best cloud model any time soon, but it does not need to. If it handles 80 to 90% of everyday tasks instantly, privately and at near zero marginal cost, that is a huge win. Reserve the cloud for the genuinely hard requests, not every prompt.
also this is just the beginning. give it 5-10 years and it may be more like 99%.
Do you use cloud or local ai to write your comments?
Most people don’t need deep agentic ai on their phones, they just need quick answers to questions, to add events to their calendars, answer emails, and remember things about their lives. These local llms actually perform better than the cloud ones for these tasks
Yeah, basically an enhanced Siri.
He can see into the future.
Future #1: The lifetime of the pure AI companies is limited.
Future #2: Local LLM’s are trending, because “token” prices will go up like crazy.
The present: Open Weight AI, such as Kimi’s, is already almost exactly as good as ClosedAI from Anthropic and »OpenAI«.
How much compute do you think you need to run kimi locally at mythos fable levels? That ain’t going to work unless people start buying small data centers locally.
About 4+ maxed M4 Studios, I guess. But that‘s not the point: in 80%+ of cases, people won’t need that kind of AI model to solve their problems.
So a non viable non starter. Got it
Yeah “See into the future”. They’ve done the business analysis and determined the outcomes you gave. Apple has the confidence as a company to hold back even if the market wants them to do something and know they can hold firm through the irrationality of the market and come out the other side.
One has to wonder how companies like Google and Microsoft will fare when the financial engineering blows up in their faces. I’m guessing they think the government will bail them out.
Apple has always been unusually willing to sacrifice short term hype for long term positioning. That does not guarantee they are right, but it is a very different bet from spending hundreds of billions assuming demand will eventually justify the buildout. If AI demand disappoints, discipline suddenly looks a lot more valuable than scale.
This isn’t seeing into the future. This is just taking a look at the present.
Present #1: AI datacenters cost a fuck ton
Present #2: No customer is willing to pay the costs of AI unless they sell at a loss
Tim Cook for all his faults is always been great at politics and this is an extension of that.
Cook’s biggest product might be expectation management. He rarely promises tomorrow’s miracle, which buys Apple room to ship when it suits them instead of when Wall Street gets impatient.
It seems to have been a plan for a long time, given their huge shift to unified memory architectures across most of their hardware.
They’re pretty much the only vendor where you can cost-effectively deploy a foundational LLM locally.
It would seem so. On the other hand, it is puzzling that they did not also allocate the necessary resources to the development of LLMs. 🤷
Why bother? Just wait for one of the AI bro companies to go bust and buy the wreckage.
Hehe… 😈
Still nobody build a unified memory model with HBM. I would buy Apple hardware, then. If I can put Linux on it.
Perhaps I can buy a used Instict with generics science acceleration, once the smoke clears. And add a couple solar kWp to my capacity.
Looks like an interesting approach
Absolutely. Looking forward to seeing the next generation of Macs.
I couldn’t give a shit less about apple but man I’m I hoping this comes to fruition.
I’d like to buy hardware again.
And this will have huge ramifications for my field, energy, because a substantial portion of compute power usage will move out of data centers and into the edge (your iPhone)
The decentralised operation of LLMs would also be significantly simpler and cheaper for the use of decentralised renewable energy sources.
At least gpus will finally cease their breakneck price climb, I say, for the 800th time over the last two decades.
Apple’s AI strategy is leaving them behind
Apple’s stock drops on failure to meet AI promises
Etc.
Now whose stock is dropping?
Stock prices aren’t proof of being right, but they do show investors can change their minds a lot faster than the narratives do.
I am practicing that strategy now. I recently upgraded to an iPad Pro M5 (the day price increases were announced, jumped on a deal immediately), upgraded several shortcuts with Apple Intelligence, and am refining them to run entirely on-device instead of in PCC (not using ChatGPT at all).
Just a note that PCC is different from the ChatGPT integration. You might be using PCC resources without noticing, because the iPad won’t tell you in advance. Only way to be sure is to disconnect from the Internet and/or check your Apple Intelligence Report (in System Settings > Privacy) after the fact
I’m referring to my Shortcuts where I can specify on-device or cloud (PCC).
You can’t. iPadOS decides for you whether a request is run locally or in the cloud.
I’m waiting for the next generation of Mac Mini and Mac Studio.
Tim Cook will say anything to try and get ahead. First Apple’s talking about how good their AI is/will be, but now fears of over-hyped AI spending and data center issues, etc, now he’s claims oh and it’ll all run locally. Apples typical “we’re not the bad guys”
While local LLMs are getting better, I still feel like overall, this will tank battery life, and through several million devices charging, will still use a lot of extra power overall. Plus, a lot of the data center issues falls to training and ingesting information, so, Apple let’s the other guys do the heavy lifting for them to seem less evil? Wasn’t Apple initially involved with OpenAI when they were a nonprofit and cofunded business? Idk, I still say Apple isn’t trustworthy.
Apple’s marketing deserves skepticism, but the technical argument is separate. Local inference does not eliminate giant training clusters, it mainly cuts inference costs, latency and improves privacy. Apple still uses cloud models when needed.
