DBRX Instruct is a large language model that employs a mixture-of-experts (MoE) approach, developed from the ground up by Databricks. It is designed to excel at interactions that span a few turns.
DBRX Instruct Model Overview
DBRX Instruct is a decoder-only, transformer-based large language model with 132 billion total parameters, integrating a detailed mixture-of-experts (MoE) framework. Of its total parameters, 36 billion are activated for any given input.
This model stands out by employing a fine-grained MoE architecture, setting it apart from other open MoE models like Mixtral-8x7B and Grok-1 by utilizing a larger quantity of smaller experts—specifically, 16 experts with 4 chosen per input compared to the 8 experts with 2 chosen in the aforementioned models.
This approach allows for 65 times more combinations of experts, significantly enhancing model quality. DBRX incorporates advanced features such as rotary position encodings (RoPE), gated linear units (GLU), and grouped query attention (GQA), utilizing the GPT-4 tokenizer from the tiktoken repository. These features were selected based on comprehensive evaluations and scaling studies.
Pretrained on 12 trillion tokens of meticulously selected text and code data, with a maximum context length of 32,768 tokens, the dataset for DBRX is considered to be twice as effective, token-for-token, as those used in training previous models like the MPT family. The dataset benefits from Databricks’ complete toolset, including Apache Spark™ and Databricks notebooks for data processing, along with Unity Catalog for data management and governance. The pretraining phase also incorporated curriculum learning, adjusting the data mixture to significantly enhance model quality.
DBRX Instruct is tailored for text-based inputs and outputs, accepting up to 32,768 tokens in context length. For more in-depth information on DBRX Instruct and DBRX Base, interested individuals are directed to the technical blog post by the developers. The model is released under the Databricks Open Model License, adhering to the Databricks Open Model Acceptable Use Policy, and is currently at version 1.0, owned by Databricks, Inc.
Usage
The DBRX models, including DBRX Base and DBRX Instruct, offer versatile applications and can be accessed through multiple platforms:
- HuggingFace Access: DBRX Base and DBRX Instruct are downloadable from HuggingFace. The HuggingFace repository houses both models, providing an easy path for users to engage with the models directly.
- GitHub Repository: The models are also hosted on GitHub, where users can find the DBRX model repository for additional resources, updates, and community contributions.
- Databricks Foundation Model APIs: For enterprise deployments, DBRX Base and DBRX Instruct are integrated into the Databricks Foundation Model APIs. These APIs support both Pay-per-token and Provisioned Throughput options, catering to different usage scales and preferences.
- Fine-tuning Documentation: For users interested in customizing the models further, detailed guides on how to fine-tune using the LLM-Foundry are available here. This documentation covers both pretraining and fine-tuning processes, aiding users in tailoring the models to specific needs or projects.
These pathways offer broad access and customization options, ensuring that users can effectively utilize DBRX Instruct models in various scenarios and for different applications.
DBRX Instruct Training Dataset Limitations
The training dataset for the DBRX models, comprising 12 trillion tokens of text, has its strengths but also notable limitations:
- Knowledge Cutoff: The models’ information is current up to December 2023. Beyond this date, DBRX may not have awareness of more recent events, developments, or data.
- Content Type: The dataset includes a mix of natural-language texts and code examples, supporting the model’s ability to understand and generate both types of content effectively.
- Language Proficiency: Predominantly, the training data is in English, making DBRX primarily proficient in English language tasks. Its capabilities in understanding or generating non-English content were not extensively tested, suggesting a potential limitation in performance with languages other than English.
- Modality: DBRX is designed for text-based inputs and outputs only and lacks multimodal capabilities. This means it cannot process or generate content in forms other than text, such as images or audio.
These aspects define the scope and potential application areas for DBRX, indicating it as a powerful tool for English language and coding tasks, while also highlighting areas where it may not be the best fit without additional development or integration with other models or systems.
Read related articles: