MiDashengLM-7B: The Open-Source Audio Model Powering Tomorrow’s Smart World

MiDashengLM-7B is an open-source, 7-billion-parameter large language model designed specifically for understanding and reasoning over general audio data. Unlike many peer models, which focus primarily on text or images, MiDashengLM-7B is all about sound — including music, speech, and environmental audio. It forms the backbone of Xiaomi’s intelligent audio capabilities in smart homes and connected vehicles.

Key ambitions of the model include:

• Superior general audio understanding: Excels at captioning and classifying a wide range of audio types.
• Open source & reproducible: Built with published training data and pipeline, fostering transparency and collaboration.
• Highly efficient: Radical improvements in inference speed, batch processing, and memory efficiency.

Technical Components & Architecture

1. Foundation

• Decoder: Built upon the Qwen2.5-Omni-7B Thinker, a robust, next-generation transformer decoder.
• Audio Encoder: Uses the Dasheng encoder, which has set benchmarks for universal audio representation.

2. Novel Caption-Based Alignment

Instead of training only on automatic speech recognition (ASR) transcripts, MiDashengLM-7B employs general audio captions as training targets. This exposes the model to richer semantic context and enables true multi-domain audio understanding.

3. Model Pipeline

• Audio Input → Dasheng Encoder: Extracts robust, context-rich audio features.
• Caption-Aligned Decoder (Qwen2.5-Omni-7B): Generates responses or classifies content based on encoded audio representations.
• Unified Pipeline: Enables batch inference of up to 512 audio samples (30s each) on an 80GB GPU, a feat unmatched by competitors.

Novelities and Innovations

• Ultra-Efficient Inference: Achieves up to 20× speedup over industry peers for large batch sizes, and a 3.2× speedup at batch sizes where other models plateau or fail (e.g., OOM at batch >8 for competitors).
• Minimal First-Token Delay: Delivers audio-to-text responses with first-token latency at just 25% of what comparable models achieve.
• Deployment-Ready: Designed for offline device use, opening up AI for edge applications in cars, smart homes, and mobile devices.
• Open Data & License: Trained with full transparency and released under a permissive, business-friendly license

Benchmark Results

Code Implementation

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "mispeech/midashenglm-7b"

model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

user_prompt = "Caption the audio."  # You may try any other prompt 

messages = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": "You are a helpful language and speech assistant."}
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "text", "text": user_prompt},
            {
                "type": "audio",
                "path": "/path/to/example.wav",
                # or "url": "https://example.com/example.wav"
                # or "audio": np.random.randn(16000)
            },
        ],
    },
]
with torch.no_grad():
    model_inputs = processor.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        add_special_tokens=True,
        return_dict=True,
    )
    generation = model.generate(**model_inputs)
    output = processor.batch_decode(generation, skip_special_tokens=True)  # ["An engine is idling."]

MiDashengLM-7B: The Open-Source Audio Model Powering Tomorrow’s Smart World was originally published in Data Science in Your Pocket on Medium, where people are continuing the conversation by highlighting and responding to this story.