Elon Musk's xAI company released its large language model Grok as “open source” over the weekend. The billionaire is clearly hoping to put his company at odds with rival OpenAI, which despite its name is not particularly open. But does releasing code for something like Grok actually contribute to the AI development community? Yes and no.
Grok is a chatbot trained by xAI to fulfill the same vaguely defined role as something like ChatGPT or Claude: you ask it, it answers. However, the LLM was given a brazen tone and additional access to Twitter data as a way to set it apart from the rest.
As always, it's almost impossible to evaluate these systems, but the general consensus seems to be that they're competitive with mid-range models from the last generation like GPT-3.5. (Whether you decide this is impressive given the short development time frame or disappointing given the budget and hype surrounding xAI is entirely up to you.)
In any case, Grok is a modern, functional LLM of great size and capabilities, and the more access the developer community has to the heart of such things, the better. The problem is defining “open” in a way that does more than just allow a corporation (or billionaire) to claim the moral high ground.
This is not the first time that the terms “open” and “open source” have been questioned or misused in the world of AI. And we're not just talking about a technical issue, like choosing a non-open-use license like any other (Grok is Apache 2.0, if you're wondering).
First of all, AI models are different from other software when it comes to being “open source”.
If you make, say, a word processor, it's relatively easy to make it open source: you publish all your code publicly and allow the community to suggest improvements or create their own version. Part of what makes open source as a concept valuable is that every aspect of the application is original or attributed to its original creator — this transparency and commitment to proper attribution is not just a byproduct, but the core of the concept of openness itself.
With AI, this is arguably not possible at all, because the way machine learning models are created involves a largely unknown process where a huge amount of training data is distilled into a complex statistical representation that is not really guided by any human, or even He understands it. . This process cannot be inspected, audited, and optimized in the way that traditional code can – so although it still has tremendous value in one sense, it can never be truly open. (The standards community has not even defined what open domain is in this context, but is actively discussing it.)
But that hasn't stopped AI developers and companies from designing their models and claiming them as “open,” a term that has lost much of its meaning in this context. Some call their model “open” if there is a public interface or application programming interface (API). Some call it “open” if they release a paper describing the development process.
Arguably the closest an AI model comes to being “open source” is when its developers release its weights, i.e. the precise features of the countless nodes of its neural networks, which perform vector mathematical operations in precise order to complete the model initiated by user input. But even “open-weighted” models like LLaMa-2 leave out other important data, such as the training and process dataset, which would be necessary to recreate them from scratch. (Some projects go further than that, of course.)
All of this is before we mention the fact that it takes millions of dollars of computing and engineering resources to create or replicate these models, effectively limiting who can create and replicate them to companies with deep pockets.
So where does Grok's version of xAI fall on this spectrum?
As an open source model, it is ready for anyone to download, use, modify, enhance or distill. This is good! It appears to be among the largest models anyone can freely access this way, in terms of parameters – 314 billion – giving curious engineers a lot to work with if they want to test how it performs after various tweaks.
The size of the model has a serious drawback, though: you'll need hundreds of gigabytes of high-speed RAM to use it in this prototype. If you don't already have, say, a dozen Nvidia H100s in your six-figure AI inference machine, don't bother clicking that download link.
And while the Grok is arguably competitive with some other modern models, it's also much larger, which means it requires more resources to accomplish the same thing. There's always a hierarchy of size, efficiency and other metrics, and that's still valuable, but it's more of a raw material than a finished product. It's also not clear if this is the latest and greatest version of Grok, like the obviously tuned version that some have access to via X.
Overall, publishing this data is a good thing, but it's not as game-changing as some had hoped.
It's also hard not to wonder why Musk would do this. Is his AI startup really dedicated to open source development? Or is this just clay in the eye of OpenAI, with which Musk is currently seeking billionaire-level beef?
If they are truly committed to open source development, this will be the first of many releases, and hopefully they will take community feedback into account, release other important information, characterize the training data process, and explain their approach further. If it's not, and it's just so Musk can point it out in online arguments, it will still be valuable — but it's not something anyone in the AI world will rely on or care much about beyond the next few months while playing with it. Model.
