Machine Learning has iconic applications in programming languages, from code understanding to code representation or completion. Previous work has focused on exploiting the underlying deep semantic structure of programming languages ​​such as Code2Vec, Code2Seq, and Graph Representation Learning for Code. The above architectures are adapted to the native structures of Abstract Syntax Trees (AST) / Data Flow Graphs (DFG). They have one major limitation: they can only be applied to tasks that involve fully executable code.

Later research showed how transformer-based models can be used like natural language to code at the lexical level (text). Since then, language models have been widely used to model code in various tasks. Such models run every few seconds, especially in the case of code completion. Strong models running on consumer devices are preferred to avoid network latency, latency, and address discrepancies related to closed APIs.

Researchers from Stability AI presented Fixed Code, which serves as a general-purpose core code language model targeting code integration, logic, mathematics, and other software engineering-based tasks. They also introduce a command variant called Fixed Code Command which enables conversation with the model in a natural conversational interface to perform question-answering and instruction-based tasks.

Stable Code is built on top of Stable LM, a state-of-the-art LLM for natural language in English at the 3 billion parameter scale. The model is a causal decoder-only transformer similar in design to the LLaMA architecture. The main differences with LLaMA are:

• Location Integrations: Rotary position embeds are applied to the first 25% of head embed dimensions for improved performance.
• Normalization: LayerNorm with learned bias terms as opposed to RMSNorm.
• Prejudices: All bias terms were removed from the feed-forward networks and multi-head self-attention levels, except for the biases of the key, query, and value views.

The stable code matches the performance of Llama and StarCoder on average across all programming languages, although it is relatively smaller. Also, Stable Code 3B achieves strong performance on the 3B scale, showing remarkable capabilities in code completion tasks. They also evaluated instruction-tuned models on the challenging Multi-turn benchmark code subset.

In conclusion, the researchers from Stability AI introduced Stable Code and Stable Code Instruct to address different software development use cases. Both Stable Code and Stable Code Instruct are compact decoder-only language models. Researchers have performed extensive model evaluations and comparisons with other similarly sized models, demonstrating the remarkable performance of Stable Code and Stable Code Instruct. They also provide a breakdown of the model on typical state-of-the-art computer architectures.

check it Paper and Blog. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us Twitter. Join us Telegram channel, Discord Channeland LinkedIn Groops.

If you like our work, you will love our work newsletter..

Don’t forget to join us 39k+ ML SubReddits