Current "good enough" models like Mistral Small require GPUs like the RTX 6000 to achieve user-friendly response times. The model quality is good enough, especially for narrow-scope tasks like summarization and classification.
If Moore's Law holds for a few more years, a mobile device will be able to run it on-device in around 8 years (Apple's A11: 410 GFLOPS vs. RTX 6000: 16 TFLOPS [1]).
This is under the assumption that we don't see any significant optimization in the meantime. Looking back over the last eight years, the probability of no progress on the software side is near zero.
For a breakthrough in the consumer market, running LLM on-device with today's capabilities requires solving one key topic: "JIT learning" [2]. We can see some progress here [3, 4]. Perhaps the transformer architecture is not the best for this requirement, but it is hard to argue that it is impossible for Generative AI.
Due to today's technical limitations, we don't have real personal assistants. This could be the Mac for Apple in the AI era.
Isn't "Private Compute Cloud" just a marketing term with some restrict sec architecture? The real personal assistant LLM would mean to have the realtime data available in hot memory (to make sure to give instant responses).
Audio, video, screen recordings, etc. from a single customer could be something between 1 and 10 GByte per day on average. After training you might get something like 3 MByte in additional model size per day. Even with 1 billion active users you would need to store additional data with 1 billion GByte (again on hot storage, like expensive GPU memory). The total amount of the memory of GPUs sold by NVIDIA is not even close to 400mio GByte (NVIDIA 3.8mio data center GPUs in 2023).
Google AI summaries isn't a chatbot exactly, but probably has been successful in staving off migration to chatgpt search, at least once it improved a lot.
- If you get into a car crash with your iPhone a ML model detects this and automatically calls emergency services.
- If you are wearing an Apple Watch a ML model is constantly analyzing your heart rhythm and will alert you to (some types of) irregularities. It's so computationally efficient it can literally do this in the background all day long.
- When you take any picture on any iPhone a whole array of ML models immediately run to improve the image. More models are used when manually editing images.
- After you save the photo ML models run to analyze and index the photo so it's easily searchable later. That's why you can search for "golden retriever" and get actual results.
- When you speak at your device (for example, to dictate a text message) there's a ML model that transcribes that into text. Likewise, when you're hands-free and want to hear an incoming text message, an ML model converts it to audio. All on-device and available offline at that.
Or are we playing that stupid game where "AI === LLM"?
It looks like the parent was asking about LLMs specifically, in which case I don't think those two count. AFAIK Adobe's image-generation stuff is a diffusion model, not an LLM, and Nvidia's DLSS isn't an LLM either.
I have to admit - outside of cost and core values differences - Adobe has been an early player and often overlooked - but the company smells so bad that I guess they're a little looking for it.
> there was a smartphone before the iPhone; there were many tablets before the iPad; there was an MP3 player before iPod
That's the biggest shift I've heard from Apple. They were either "first" or ignored the existence of competing features/products for ages. I'm really surprised by this quote.
Compare "smartphones before iPhone" to the original announcement:
> iPhone also ushers in an era of software power and sophistication never before seen in a mobile device, which completely redefines what users can do on their mobile phones. (...) iPhone is a revolutionary and magical product that is literally five years ahead of any other mobile phone,
That's actually very consistent for Apple. Apple doesn't generally claim to be the first to do something, but have always taken the line that they're the first to execute it well. Hence their fondness for words like 'reimagine', 'revolutionise', etc.
Yes, dominant smartphones before iPhone were BlackBerry inspired, full physical keyboard with small screen.
When the iPhone launched, the Android project changed direction toward a full screen phone and that form became much more dominant and popular than the BlackBerry form.
Apple made the bet that they could make the full screen experience much more compelling that people would accept the trade off of losing the keyboard.
I've heard the phrase "through Apple new technologies achieve their final form", possibly not official Apple but one of the Apple choir bloggers (Gruber?).
There were smartphones before iPhone, now all smartphones are black featureless rectangles.
There were printers before LaserWriter, then for 20 years all printers became this. (And later disappeared.)
There were wireless heaphones before Airpods, now the difference is in the shape of the stubs.
There were laptops before the Macbook Air... etc
They didn't, that's why I explicitly wrote "or ignored the existence of competing features/products for ages".
But they sure write releases like that's implied.
> iPhone introduces an entirely new user interface based on a large multi-touch display and pioneering new software, letting users control iPhone with just their fingers.
Large display existed before, "just fingers" control was... always the case, the interface was quite polished, but it existed elsewhere, etc. But if you didn't know that, reading the announcement sure sounds like it's never been done before. It's the multi touch that makes the combination novel.
I wish apple would provide a decent model to apple intelligence and let developers build on it. Like sure it would lose a lot of money right now, but it would mean that app developers making AI agents on the iphone could still charge modest amounts if they aren't responsible for the inference costs.
Chief Bean Counter Cook doesn't do cool, goodwill, or long term strategy. Only making the same set of products incrementally better and more expensive, and increasingly prone to expensive repair.
Apple has a ton of problems, but your comments don't address them (primarily the perceived decline in software quality and app store developer gouging).
While cool is subjective, what new, mass-market products should they create? Which product market should they "re-invent"? I wish they'd buy Sonos and fix that shit show, but that's not a profitable market to enter.
Barring the price jump around the iPhone 7, their smartphones have stayed about the same price. [1]
Over half of all smartphone repairs are battery replacements, which implies people don't take care of their batteries or are keeping their phones long enough to wear the battery down normally. [2] Additionally, Apple ranks very well in repairability. [3] They also support their phone's software longer on older devices than the competition.
Apple may be greedy but they can’t be accused of bungling long term strategy.
While OpenAI sells $2 bills for $1, Tim Cook was out there increasing service revenue and profitability so that it was larger than Macs and iPads combined.
Tim Cook presided over some incredibly lucrative product launches like AirPods, TV+, Apple Music, moved chip design in house which doubled Mac market share and has made the iPhone continually dominant, they’ll even drop third party 5G models soon. These are all incredibly shrewd long term strategy moves.
Yeah but they're just too small to do anything useful with yet. Like we're in this weird state where you can't easily sell usage based pricing through appstore payments (and customers don't really understand usage anyways). So you need to sell access to an agent via a subscription, but your costs are 90% usage based so it's hard to price. If appstore developers could use a quota of access tokens from a users apple intelligence subscription we could offer AI agents for $3-5/mo and they would be actually usable! But if you need to pay for inference costs it has to be $10-20/mo. It's just a lame experience and makes the web the place to build agents even though they'd be more useful on mobile devices.
They're just too small though, maybe in the future you will be able to run a larger model on device or smaller models will become more finetuneable to be actually useful, but the current slate are pointless.
I don’t know if Tim Cook has actually tried Siri day-to-day like most typical users who get frustrated. Half the time I ask siri to convert units of measure, it pulls up a web search? We’re not asking for siri to have LLM abilities to compose sonnets. Interestingly, does anyone remember when siri would use WolframAlpha for answers? Perhaps that’s the company Apple should buy not OpenAI.
Tim Cook is a "Keep things ticking along" CEO, not a "Change course to a new destination" CEO. Initiatives like this will probably require different leadership to succeed.
Exactly, Tim Cook is a finance guy - he knows the numbers and how to keep Apple profitable. What he lacks is product vision. His one opportunity (ironically, vision) fell flat.
Don’t downvote this guy! Interesting to note that he probably constantly got criticized for not having vision, so he took that literally and called the product Vision Pro.
It’s the kind of mistake an LLM would make. Very Lacanian.
I both don't disagree with this really, but also, as an ops person, yes it is crazy hard building some of the most micro miniature systems on the planet and having someone who can see to the details of production is a pretty vital skill.
Still not a customer facing / product development role. At the same time though, again, so much of what makes Apple's products so good is that they have been amazing at product having to work with manufacturing to push the bounds of what is possible. Apple Vision for example taps this intersection: part of the product very much was figuring out physically what it was you could build.
(Something about the past year has really really shifted my perspective, enhanced the already huge respect I have for people making physical things.)
Readit News
This is under the assumption that we don't see any significant optimization in the meantime. Looking back over the last eight years, the probability of no progress on the software side is near zero.
For a breakthrough in the consumer market, running LLM on-device with today's capabilities requires solving one key topic: "JIT learning" [2]. We can see some progress here [3, 4]. Perhaps the transformer architecture is not the best for this requirement, but it is hard to argue that it is impossible for Generative AI.
Due to today's technical limitations, we don't have real personal assistants. This could be the Mac for Apple in the AI era.
[1] https://gadgetversus.com/graphics-card/apple-a11-bionic-gpu-...
[2] Increasing context size is not a valid option for my scenario as it also increases the computation demand linear.
[2] https://arxiv.org/abs/2311.06668
[3] https://arxiv.org/abs/2305.18466
[Edit: decimal separator mess]
Audio, video, screen recordings, etc. from a single customer could be something between 1 and 10 GByte per day on average. After training you might get something like 3 MByte in additional model size per day. Even with 1 billion active users you would need to store additional data with 1 billion GByte (again on hot storage, like expensive GPU memory). The total amount of the memory of GPUs sold by NVIDIA is not even close to 400mio GByte (NVIDIA 3.8mio data center GPUs in 2023).
To be clear, just having a chatbot website/app does not count.
- If you are wearing an Apple Watch a ML model is constantly analyzing your heart rhythm and will alert you to (some types of) irregularities. It's so computationally efficient it can literally do this in the background all day long.
- When you take any picture on any iPhone a whole array of ML models immediately run to improve the image. More models are used when manually editing images.
- After you save the photo ML models run to analyze and index the photo so it's easily searchable later. That's why you can search for "golden retriever" and get actual results.
- When you speak at your device (for example, to dictate a text message) there's a ML model that transcribes that into text. Likewise, when you're hands-free and want to hear an incoming text message, an ML model converts it to audio. All on-device and available offline at that.
Or are we playing that stupid game where "AI === LLM"?
Nvidia
Also, it sounds like Cook and Federighi just repeated talking points the public has already heard, so I'm not sure what the point of this was.
If there are any current Apple employees here, maybe they can weigh in.
Unless things have changed in the last 15 years, my understanding was that they actually are barred from doing just that
Though it sounds like what they actually got was fairly in-substantive statements without a clearly articulated AI strategy.
Doesn't mean that Apple doesn't have a promising AI strategy though, if so, it wasn't communicated in this Pep Talk: so what was the point?
Perhaps to look like they are doing something? Are empty words better than no words at all?
That's the biggest shift I've heard from Apple. They were either "first" or ignored the existence of competing features/products for ages. I'm really surprised by this quote.
Compare "smartphones before iPhone" to the original announcement:
> iPhone also ushers in an era of software power and sophistication never before seen in a mobile device, which completely redefines what users can do on their mobile phones. (...) iPhone is a revolutionary and magical product that is literally five years ahead of any other mobile phone,
When the iPhone launched, the Android project changed direction toward a full screen phone and that form became much more dominant and popular than the BlackBerry form.
Apple made the bet that they could make the full screen experience much more compelling that people would accept the trade off of losing the keyboard.
Strongly disagree with this. Their marketing often claims inventing things that have existed.
There were smartphones before iPhone, now all smartphones are black featureless rectangles. There were printers before LaserWriter, then for 20 years all printers became this. (And later disappeared.) There were wireless heaphones before Airpods, now the difference is in the shape of the stubs. There were laptops before the Macbook Air... etc
I recall marketing comparing iPhones to blackberries. They even had iTunes running on Motorola phone
https://www.makeuseof.com/itunes-phone-before-the-iphone-exp...
Nobody claimed Apple was the first at this. They were just the best, eventually. But it’s been 20 years
But they sure write releases like that's implied.
> iPhone introduces an entirely new user interface based on a large multi-touch display and pioneering new software, letting users control iPhone with just their fingers.
Large display existed before, "just fingers" control was... always the case, the interface was quite polished, but it existed elsewhere, etc. But if you didn't know that, reading the announcement sure sounds like it's never been done before. It's the multi touch that makes the combination novel.
There were smartphones before the iPhone. Consider the IPAQ and Windows Mobile 6.0.
And of course plenty of MP3 players before iPod.
While cool is subjective, what new, mass-market products should they create? Which product market should they "re-invent"? I wish they'd buy Sonos and fix that shit show, but that's not a profitable market to enter.
Barring the price jump around the iPhone 7, their smartphones have stayed about the same price. [1]
Over half of all smartphone repairs are battery replacements, which implies people don't take care of their batteries or are keeping their phones long enough to wear the battery down normally. [2] Additionally, Apple ranks very well in repairability. [3] They also support their phone's software longer on older devices than the competition.
[1] https://www.androidauthority.com/iphone-price-history-322149... [2] https://www.businessresearchinsights.com/market-reports/smar... [3] https://www.ifixit.com/repairability/smartphone-repairabilit...
While OpenAI sells $2 bills for $1, Tim Cook was out there increasing service revenue and profitability so that it was larger than Macs and iPads combined.
Tim Cook presided over some incredibly lucrative product launches like AirPods, TV+, Apple Music, moved chip design in house which doubled Mac market share and has made the iPhone continually dominant, they’ll even drop third party 5G models soon. These are all incredibly shrewd long term strategy moves.
Instead Apple can’t even manage to implement speech to text that works in safari and can’t manage to make Siri not suck.
It’s the kind of mistake an LLM would make. Very Lacanian.
Still not a customer facing / product development role. At the same time though, again, so much of what makes Apple's products so good is that they have been amazing at product having to work with manufacturing to push the bounds of what is possible. Apple Vision for example taps this intersection: part of the product very much was figuring out physically what it was you could build.
(Something about the past year has really really shifted my perspective, enhanced the already huge respect I have for people making physical things.)
Based on what exactly? He led the overhaul of a massive amount of Apple under his tenure.