Compare commits
2
Commits
ff30c3aa92
..
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6677070eef | ||
|
|
49da03b37a |
Generated
+1
@@ -0,0 +1 @@
|
|||||||
|
Rainnya
|
||||||
Generated
+8
@@ -0,0 +1,8 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<project version="4">
|
||||||
|
<component name="MarkdownSettings">
|
||||||
|
<option name="previewPanelProviderInfo">
|
||||||
|
<ProviderInfo name="Compose (experimental)" className="com.intellij.markdown.compose.preview.ComposePanelProvider" />
|
||||||
|
</option>
|
||||||
|
</component>
|
||||||
|
</project>
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# AGENTS.md — Rainnya (rainnya-chat)
|
||||||
|
|
||||||
|
Android AI 聊天客户端:Compose + Material3 连接 AstrBot WebSocket。Kotlin 2.2 / AGP 9.2 / Gradle 9.4 / Room 2.7(KSP) / OkHttp 4.12 / minSdk 28 / target 36。远端 Gitea `git.rainnya.asia/miaomiao/rainnya-chat`,分支 master。
|
||||||
|
|
||||||
|
## 构建(本机 Linux,重要)
|
||||||
|
- 命令:`sh ./gradlew :app:assembleDebug`(gradlew 无 +x,务必用 `sh ./gradlew`)
|
||||||
|
- ⚠️ **必坑**:项目 `gradle.properties` 硬编码 Windows JBR 路径 `org.gradle.java.home=C:\Program Files\...`,本机任何 Gradle 调用直接报 "Java home supplied is invalid"。已在本机 `~/.gradle/gradle.properties` 覆盖为 `org.gradle.java.home=/opt/android-studio/jbr`(JDK 21)。**不要把这个 Windows 路径写回项目文件**;新机器需同样覆盖
|
||||||
|
- 本机 SDK:`~/Android/Sdk`(platforms android-36.1、build-tools 36.0.0)自动发现,无需 local.properties
|
||||||
|
- 依赖走阿里云镜像(settings.gradle.kts),本机网络慢,首次构建下载依赖耗时较长
|
||||||
|
- 测试:`sh ./gradlew testDebugUnitTest`(仅示例单测);instrumented 需设备/模拟器
|
||||||
|
|
||||||
|
## 架构
|
||||||
|
```
|
||||||
|
data/
|
||||||
|
local/ Room(AppDatabase/ChatDao/Converters)
|
||||||
|
model/ ChatMessage/ChatSession/WsMessage
|
||||||
|
repository/ ChatRepository — 状态 + 业务逻辑
|
||||||
|
settings/ AppSettings — SharedPreferences "rainnya_prefs"
|
||||||
|
websocket/ AstrBotWsClient — OkHttp WS
|
||||||
|
ui/ chat components sessions settings navigation theme util
|
||||||
|
```
|
||||||
|
|
||||||
|
## WS 协议(AstrBot)
|
||||||
|
- 连接:`ws(s)://<server>/api/v1/chat/ws?api_key=<key>`(AppSettings.wsUrl 生成;cleartext http 已放行)
|
||||||
|
- 发送:`{t:"send", message, username:"app_<名>", session_id, message_id, enable_streaming:true}`
|
||||||
|
- 接收:JSON `{type, data, session_id, message_id, streaming, code, attachment_id, url}`;`type=="pong"` 忽略;流式靠 `streaming` 字段 + 段数据渲染
|
||||||
|
- 设置项存 SharedPreferences:server_url(默认 `http://192.168.1.100:6185`)/ api_key / username(自动加 `app_` 前缀)
|
||||||
|
- 本机 AstrBot 开发实例:`/home/miaomiao/Project/astrbot`,`./start.sh` 管理(凭据见项目 MEMORY.md / RainMood 记忆)
|
||||||
|
|
||||||
|
## 备注
|
||||||
|
- ⚠️ **安卓签名密钥**:签名已接入 release(build.gradle.kts signingConfig 读 `app/keystore.properties`,密钥文件 `app/rainnya-release.p12` 已 gitignore)。**真实密钥备份在 `/run/media/miaomiao/miao/备份/apkkey/riannyachat/AAA`(PKCS#12,别名 `key0`)**;store/key 密码等凭据**见项目记忆 `~/.local/share/opencode/memory/projects/rainnya-chat/MEMORY.md`,勿写进本仓库**。`app/keystore.properties` 已移出 git 追踪(防真实密码入库),`build.gradle.kts` 的 signingConfig 也确认无明文密码
|
||||||
|
- 提交信息风格:最近为单字「喵」(自动提交)+ 少量 `feat:`/`fix:`/`refactor:` 前缀
|
||||||
|
- UI 强调 API28 兼容(键盘/IME 检测用 ViewTreeObserver 兜底,勿退回仅 WindowInsets)
|
||||||
@@ -1 +1,3 @@
|
|||||||
/build
|
/build
|
||||||
|
/keystore.properties
|
||||||
|
/*.p12
|
||||||
|
|||||||
@@ -1,9 +1,17 @@
|
|||||||
|
import java.util.Properties
|
||||||
|
|
||||||
plugins {
|
plugins {
|
||||||
alias(libs.plugins.android.application)
|
alias(libs.plugins.android.application)
|
||||||
alias(libs.plugins.kotlin.compose)
|
alias(libs.plugins.kotlin.compose)
|
||||||
alias(libs.plugins.com.google.devtools.ksp)
|
alias(libs.plugins.com.google.devtools.ksp)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
val keystoreProperties = Properties()
|
||||||
|
val keystorePropertiesFile = rootProject.file("app/keystore.properties")
|
||||||
|
if (keystorePropertiesFile.exists()) {
|
||||||
|
keystoreProperties.load(keystorePropertiesFile.inputStream())
|
||||||
|
}
|
||||||
|
|
||||||
android {
|
android {
|
||||||
namespace = "com.rainnya.chat"
|
namespace = "com.rainnya.chat"
|
||||||
compileSdk {
|
compileSdk {
|
||||||
@@ -22,8 +30,18 @@ android {
|
|||||||
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
|
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
signingConfigs {
|
||||||
|
create("release") {
|
||||||
|
keyAlias = keystoreProperties.getProperty("keyAlias")
|
||||||
|
keyPassword = keystoreProperties.getProperty("keyPassword")
|
||||||
|
storeFile = keystoreProperties.getProperty("storeFile")?.let(::file)
|
||||||
|
storePassword = keystoreProperties.getProperty("storePassword")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
buildTypes {
|
buildTypes {
|
||||||
release {
|
release {
|
||||||
|
signingConfig = signingConfigs.getByName("release")
|
||||||
optimization {
|
optimization {
|
||||||
enable = false
|
enable = false
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,4 +0,0 @@
|
|||||||
storeFile=rainnya.jks
|
|
||||||
storePassword=rainnya123
|
|
||||||
keyAlias=rainnya
|
|
||||||
keyPassword=rainnya123
|
|
||||||
@@ -3,6 +3,7 @@
|
|||||||
xmlns:tools="http://schemas.android.com/tools">
|
xmlns:tools="http://schemas.android.com/tools">
|
||||||
|
|
||||||
<uses-permission android:name="android.permission.INTERNET" />
|
<uses-permission android:name="android.permission.INTERNET" />
|
||||||
|
<uses-permission android:name="android.permission.RECORD_AUDIO" />
|
||||||
|
|
||||||
<application
|
<application
|
||||||
android:networkSecurityConfig="@xml/network_security_config"
|
android:networkSecurityConfig="@xml/network_security_config"
|
||||||
|
|||||||
@@ -5,25 +5,43 @@ import androidx.room.Database
|
|||||||
import androidx.room.Room
|
import androidx.room.Room
|
||||||
import androidx.room.RoomDatabase
|
import androidx.room.RoomDatabase
|
||||||
import androidx.room.TypeConverters
|
import androidx.room.TypeConverters
|
||||||
|
import androidx.room.migration.Migration
|
||||||
|
import androidx.sqlite.db.SupportSQLiteDatabase
|
||||||
import com.rainnya.chat.data.model.ChatMessage
|
import com.rainnya.chat.data.model.ChatMessage
|
||||||
import com.rainnya.chat.data.model.ChatSession
|
import com.rainnya.chat.data.model.ChatSession
|
||||||
|
|
||||||
@Database(entities = [ChatSession::class, ChatMessage::class], version = 2, exportSchema = false)
|
@Database(entities = [ChatSession::class, ChatMessage::class], version = 3, exportSchema = false)
|
||||||
@TypeConverters(Converters::class)
|
@TypeConverters(Converters::class)
|
||||||
abstract class AppDatabase : RoomDatabase() {
|
abstract class AppDatabase : RoomDatabase() {
|
||||||
abstract fun chatDao(): ChatDao
|
abstract fun chatDao(): ChatDao
|
||||||
|
|
||||||
companion object {
|
companion object {
|
||||||
|
/** v1→v2 未做任何表结构变更,空迁移(保证后续 2→3 有完整迁移链) */
|
||||||
|
val MIGRATION_1_2 = object : Migration(1, 2) {
|
||||||
|
override fun migrate(db: SupportSQLiteDatabase) {
|
||||||
|
// 无 schema 变更
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** v2→v3:chat_messages 增加 toolCall TEXT 列(工具调用结构化数据,nullable) */
|
||||||
|
val MIGRATION_2_3 = object : Migration(2, 3) {
|
||||||
|
override fun migrate(db: SupportSQLiteDatabase) {
|
||||||
|
db.execSQL("ALTER TABLE chat_messages ADD COLUMN toolCall TEXT")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
@Volatile
|
@Volatile
|
||||||
private var INSTANCE: AppDatabase? = null
|
private var INSTANCE: AppDatabase? = null
|
||||||
|
|
||||||
fun getInstance(context: Context): AppDatabase {
|
fun getInstance(context: Context): AppDatabase {
|
||||||
return INSTANCE ?: synchronized(this) {
|
return INSTANCE ?: synchronized(this) {
|
||||||
INSTANCE ?: Room.databaseBuilder(
|
INSTANCE ?: Room.databaseBuilder(
|
||||||
context.applicationContext,
|
context.applicationContext,
|
||||||
AppDatabase::class.java,
|
AppDatabase::class.java,
|
||||||
"rainnya_db"
|
"rainnya_db"
|
||||||
).fallbackToDestructiveMigration(false).build().also { INSTANCE = it }
|
)
|
||||||
|
.addMigrations(MIGRATION_1_2, MIGRATION_2_3)
|
||||||
|
.fallbackToDestructiveMigration(false).build().also { INSTANCE = it }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ data class ChatMessage(
|
|||||||
val streaming: Boolean = false,
|
val streaming: Boolean = false,
|
||||||
val attachmentId: String? = null,
|
val attachmentId: String? = null,
|
||||||
val imageUrl: String? = null,
|
val imageUrl: String? = null,
|
||||||
|
val toolCall: String? = null,
|
||||||
)
|
)
|
||||||
|
|
||||||
enum class MessageRole { USER, ASSISTANT, SYSTEM }
|
enum class MessageRole { USER, ASSISTANT, SYSTEM }
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ data class WsIncomingMessage(
|
|||||||
val code: String? = null,
|
val code: String? = null,
|
||||||
val attachment_id: String? = null,
|
val attachment_id: String? = null,
|
||||||
val url: String? = null,
|
val url: String? = null,
|
||||||
|
val chain_type: String? = null,
|
||||||
)
|
)
|
||||||
|
|
||||||
data class WsMessageSegment(
|
data class WsMessageSegment(
|
||||||
|
|||||||
@@ -4,6 +4,11 @@ import android.content.Context
|
|||||||
import android.net.Uri
|
import android.net.Uri
|
||||||
import android.util.Log
|
import android.util.Log
|
||||||
import com.google.gson.Gson
|
import com.google.gson.Gson
|
||||||
|
import com.google.gson.JsonElement
|
||||||
|
import com.google.gson.JsonNull
|
||||||
|
import com.google.gson.JsonObject
|
||||||
|
import com.google.gson.JsonParser
|
||||||
|
import com.google.gson.JsonPrimitive
|
||||||
import com.rainnya.chat.data.local.AppDatabase
|
import com.rainnya.chat.data.local.AppDatabase
|
||||||
import com.rainnya.chat.data.model.ChatMessage
|
import com.rainnya.chat.data.model.ChatMessage
|
||||||
import com.rainnya.chat.data.model.MessageRole
|
import com.rainnya.chat.data.model.MessageRole
|
||||||
@@ -17,9 +22,11 @@ import kotlinx.coroutines.CoroutineScope
|
|||||||
import kotlinx.coroutines.Dispatchers
|
import kotlinx.coroutines.Dispatchers
|
||||||
import kotlinx.coroutines.Job
|
import kotlinx.coroutines.Job
|
||||||
import kotlinx.coroutines.delay
|
import kotlinx.coroutines.delay
|
||||||
import kotlinx.coroutines.flow.Flow
|
|
||||||
import kotlinx.coroutines.flow.MutableStateFlow
|
import kotlinx.coroutines.flow.MutableStateFlow
|
||||||
|
import kotlinx.coroutines.flow.SharedFlow
|
||||||
|
import kotlinx.coroutines.flow.SharingStarted
|
||||||
import kotlinx.coroutines.flow.StateFlow
|
import kotlinx.coroutines.flow.StateFlow
|
||||||
|
import kotlinx.coroutines.flow.shareIn
|
||||||
import kotlinx.coroutines.launch
|
import kotlinx.coroutines.launch
|
||||||
import kotlinx.coroutines.runBlocking
|
import kotlinx.coroutines.runBlocking
|
||||||
import kotlinx.coroutines.withContext
|
import kotlinx.coroutines.withContext
|
||||||
@@ -30,6 +37,9 @@ import java.util.concurrent.TimeUnit
|
|||||||
|
|
||||||
private const val TAG = "RainnyaRepo"
|
private const val TAG = "RainnyaRepo"
|
||||||
|
|
||||||
|
/** 工具调用缓冲上限:超过则视为异常数据重置(防御非 JSON 文本无限累积) */
|
||||||
|
private const val MAX_TOOL_CALL_BUFFER = 64 * 1024
|
||||||
|
|
||||||
class ChatRepository(
|
class ChatRepository(
|
||||||
private val scope: CoroutineScope,
|
private val scope: CoroutineScope,
|
||||||
private val context: Context,
|
private val context: Context,
|
||||||
@@ -64,7 +74,19 @@ class ChatRepository(
|
|||||||
private val sessionMessages = mutableMapOf<String, MutableList<ChatMessage>>()
|
private val sessionMessages = mutableMapOf<String, MutableList<ChatMessage>>()
|
||||||
private var streamTimeoutJob: Job? = null
|
private var streamTimeoutJob: Job? = null
|
||||||
|
|
||||||
val wsEvents: Flow<WsEvent> = wsClient.events
|
// ===== 工具调用(tool_call / tool_call_result)=====
|
||||||
|
/** 工具调用 JSON 分片缓冲:toolCallId -> 已积累文本(多工具调用按 id 区分) */
|
||||||
|
private val toolCallBuffers = mutableMapOf<String, StringBuilder>()
|
||||||
|
|
||||||
|
/** 最近路由的 toolCallId:无 id 的后续分片继续追加到该缓冲 */
|
||||||
|
private var lastToolCallId: String? = null
|
||||||
|
|
||||||
|
/**
|
||||||
|
* WS 事件广播流(M3):底层是 Channel 单消费者流,改为 Eagerly 多观察者 SharedFlow,
|
||||||
|
* ChatViewModel 与 VoiceCallEngine 都能收到全部事件;replay=0 只广播订阅之后的新事件。
|
||||||
|
*/
|
||||||
|
val wsEvents: SharedFlow<WsEvent> =
|
||||||
|
wsClient.events.shareIn(scope, SharingStarted.Eagerly, replay = 0)
|
||||||
|
|
||||||
init {
|
init {
|
||||||
scope.launch(Dispatchers.IO) {
|
scope.launch(Dispatchers.IO) {
|
||||||
@@ -112,10 +134,14 @@ class ChatRepository(
|
|||||||
_connectionState.value = ConnectionState.DISCONNECTED
|
_connectionState.value = ConnectionState.DISCONNECTED
|
||||||
}
|
}
|
||||||
|
|
||||||
fun sendMessage(text: String, attachmentId: String? = null, imageUrl: String? = null) {
|
/**
|
||||||
|
* 发送文本消息;未连接时返回 false(供语音引擎区分"已发送/未连接"),正常发出返回 true。
|
||||||
|
* attachmentId/imageUrl 用于图片消息(经 [sendMessageWithImage] 调用,忽略返回值即可)。
|
||||||
|
*/
|
||||||
|
fun sendMessage(text: String, attachmentId: String? = null, imageUrl: String? = null): Boolean {
|
||||||
if (_connectionState.value != ConnectionState.CONNECTED) {
|
if (_connectionState.value != ConnectionState.CONNECTED) {
|
||||||
Log.w(TAG, "Cannot send: not connected")
|
Log.w(TAG, "Cannot send: not connected")
|
||||||
return
|
return false
|
||||||
}
|
}
|
||||||
ensureSessionExists()
|
ensureSessionExists()
|
||||||
val messageId = UUID.randomUUID().toString()
|
val messageId = UUID.randomUUID().toString()
|
||||||
@@ -166,6 +192,14 @@ class ChatRepository(
|
|||||||
sessionId = sendSessionId,
|
sessionId = sendSessionId,
|
||||||
messageId = messageId,
|
messageId = messageId,
|
||||||
)
|
)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 语音通话识别出的文本走现有文本消息路径,同 session_id 延续上下文;返回是否成功发出 */
|
||||||
|
fun sendVoiceMessage(transcribedText: String): Boolean {
|
||||||
|
val trimmed = transcribedText.trim()
|
||||||
|
if (trimmed.isEmpty()) return false
|
||||||
|
return sendMessage(trimmed)
|
||||||
}
|
}
|
||||||
|
|
||||||
fun sendMessageWithImage(text: String, imageUri: Uri) {
|
fun sendMessageWithImage(text: String, imageUri: Uri) {
|
||||||
@@ -274,31 +308,50 @@ class ChatRepository(
|
|||||||
val isStreaming = msg.streaming ?: true
|
val isStreaming = msg.streaming ?: true
|
||||||
val trimmed = text.trimStart()
|
val trimmed = text.trimStart()
|
||||||
|
|
||||||
if (trimmed.startsWith("{") && trimmed.contains("chatcmpl-tool-")) {
|
when (msg.chain_type) {
|
||||||
if (isStreaming) resetStreamTimeout()
|
"tool_call" -> {
|
||||||
return
|
// 工具调用 JSON 分片(可能跨多条 plain 消息):积累直到可解析,不追加进助手文本
|
||||||
}
|
if (isStreaming) resetStreamTimeout()
|
||||||
|
accumulateToolCall(text)
|
||||||
|
}
|
||||||
|
"tool_call_result" -> {
|
||||||
|
// 工具调用结果:关联回对应工具调用消息
|
||||||
|
if (isStreaming) resetStreamTimeout()
|
||||||
|
handleToolCallResult(text)
|
||||||
|
}
|
||||||
|
"reasoning" -> {
|
||||||
|
// 推理链:不显示、不落库、不 TTS
|
||||||
|
if (isStreaming) resetStreamTimeout()
|
||||||
|
}
|
||||||
|
else -> {
|
||||||
|
// 无 chain_type(或未知):保留原有文本逻辑 + chatcmpl-tool- 启发式兜底
|
||||||
|
if (trimmed.startsWith("{") && trimmed.contains("chatcmpl-tool-")) {
|
||||||
|
if (isStreaming) resetStreamTimeout()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
val list = _messages.value.toMutableList()
|
val list = _messages.value.toMutableList()
|
||||||
val lastAssistant = list.indexOfLast { it.role == MessageRole.ASSISTANT }
|
val lastAssistant = list.indexOfLast { it.role == MessageRole.ASSISTANT }
|
||||||
|
|
||||||
if (isStreaming && lastAssistant >= 0 && list[lastAssistant].streaming) {
|
if (isStreaming && lastAssistant >= 0 && list[lastAssistant].streaming) {
|
||||||
list[lastAssistant] = list[lastAssistant].copy(
|
list[lastAssistant] = list[lastAssistant].copy(
|
||||||
content = list[lastAssistant].content + text
|
content = list[lastAssistant].content + text
|
||||||
)
|
)
|
||||||
_messages.value = list
|
_messages.value = list
|
||||||
resetStreamTimeout()
|
resetStreamTimeout()
|
||||||
} else {
|
} else {
|
||||||
list.add(
|
list.add(
|
||||||
ChatMessage(
|
ChatMessage(
|
||||||
id = UUID.randomUUID().toString(),
|
id = UUID.randomUUID().toString(),
|
||||||
content = text,
|
content = text,
|
||||||
role = MessageRole.ASSISTANT,
|
role = MessageRole.ASSISTANT,
|
||||||
sessionId = currentSessionId ?: "",
|
sessionId = currentSessionId ?: "",
|
||||||
streaming = isStreaming,
|
streaming = isStreaming,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
_messages.value = list
|
_messages.value = list
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
"image" -> {
|
"image" -> {
|
||||||
@@ -322,6 +375,7 @@ class ChatRepository(
|
|||||||
}
|
}
|
||||||
"end" -> {
|
"end" -> {
|
||||||
streamTimeoutJob?.cancel()
|
streamTimeoutJob?.cancel()
|
||||||
|
resetToolCallBuffers()
|
||||||
val list = _messages.value.toMutableList()
|
val list = _messages.value.toMutableList()
|
||||||
val lastAssistant = list.indexOfLast { it.role == MessageRole.ASSISTANT }
|
val lastAssistant = list.indexOfLast { it.role == MessageRole.ASSISTANT }
|
||||||
if (lastAssistant >= 0) {
|
if (lastAssistant >= 0) {
|
||||||
@@ -368,6 +422,228 @@ class ChatRepository(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ===== 工具调用(chain_type: tool_call / tool_call_result)=====
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 工具调用解析结果(规范化中间结构)。AstrBot 实际发出的紧凑格式为
|
||||||
|
* `{"id","name","args","ts"}`;OpenAI 格式为 `{"id","type":"function","function":{"name","arguments"}}`,
|
||||||
|
* 两者都会被解析并归一化。ts 统一存毫秒。
|
||||||
|
*/
|
||||||
|
private data class ParsedToolCall(
|
||||||
|
val id: String,
|
||||||
|
val name: String?,
|
||||||
|
val args: String?,
|
||||||
|
val result: String?,
|
||||||
|
val ts: Long?,
|
||||||
|
val finishedTs: Long?,
|
||||||
|
)
|
||||||
|
|
||||||
|
/** 未知 id 分片暂存 key(首个分片尚未带 id 时) */
|
||||||
|
private val TOOL_CALL_PENDING_KEY = "__pending__"
|
||||||
|
|
||||||
|
/** 工具调用 JSON 中 id 字段的正则(仅用于分片路由,完整解析靠 Gson) */
|
||||||
|
private val TOOL_CALL_ID_REGEX = Regex(""""id"\s*:\s*"([^"]+)"""")
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 积累一条工具调用分片(可能跨多条 plain 消息,如 arguments 分段下发)。
|
||||||
|
* 按 id 区分缓冲;无 id 的分片路由到最近活跃缓冲;解析成功后创建/更新消息。
|
||||||
|
*/
|
||||||
|
private fun accumulateToolCall(chunk: String) {
|
||||||
|
if (chunk.isBlank()) return
|
||||||
|
val id = extractToolCallId(chunk)
|
||||||
|
if (id != null) {
|
||||||
|
lastToolCallId = id
|
||||||
|
// 把 id 出现前暂存在 pending 的前缀并入该 id 缓冲(保证顺序完整)
|
||||||
|
val pending = toolCallBuffers.remove(TOOL_CALL_PENDING_KEY)
|
||||||
|
val buf = toolCallBuffers.getOrPut(id) { StringBuilder() }
|
||||||
|
if (pending != null) buf.insert(0, pending)
|
||||||
|
buf.append(chunk)
|
||||||
|
} else {
|
||||||
|
val buf = lastToolCallId?.let { toolCallBuffers[it] }
|
||||||
|
?: toolCallBuffers.getOrPut(TOOL_CALL_PENDING_KEY) { StringBuilder() }
|
||||||
|
buf.append(chunk)
|
||||||
|
}
|
||||||
|
|
||||||
|
val key = id ?: lastToolCallId ?: TOOL_CALL_PENDING_KEY
|
||||||
|
val full = toolCallBuffers[key]?.toString() ?: return
|
||||||
|
|
||||||
|
// 防御:数据不是可解析 JSON 时避免缓冲无限增长
|
||||||
|
if (full.length > MAX_TOOL_CALL_BUFFER) {
|
||||||
|
Log.w(TAG, "Tool call buffer exceeded limit, resetting")
|
||||||
|
resetToolCallBuffers()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
val parsed = tryParseToolCall(full) ?: return
|
||||||
|
upsertToolCallMessage(parsed)
|
||||||
|
// 已完整解析消费:清空缓冲,防止重复 upsert
|
||||||
|
toolCallBuffers[key]?.setLength(0)
|
||||||
|
if (key == TOOL_CALL_PENDING_KEY) {
|
||||||
|
toolCallBuffers.remove(TOOL_CALL_PENDING_KEY)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 处理工具调用结果:关联回对应工具调用消息并更新 result/finished_ts */
|
||||||
|
private fun handleToolCallResult(text: String) {
|
||||||
|
val parsed = tryParseToolCallResult(text) ?: return
|
||||||
|
toolCallBuffers.remove(parsed.id)
|
||||||
|
val list = _messages.value.toMutableList()
|
||||||
|
val idx = list.indexOfFirst { msg ->
|
||||||
|
msg.toolCall?.let { toolCallIdOf(it) == parsed.id } == true
|
||||||
|
}
|
||||||
|
if (idx < 0) {
|
||||||
|
Log.d(TAG, "Tool call result ignored: no matching call ${parsed.id}")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
val old = list[idx]
|
||||||
|
val oldJson = old.toolCall?.let { runCatching { JsonParser.parseString(it) }.getOrNull() }
|
||||||
|
if (oldJson !is JsonObject) return
|
||||||
|
if (parsed.result != null) oldJson.addProperty("result", parsed.result)
|
||||||
|
if (parsed.finishedTs != null) oldJson.addProperty("finished_ts", parsed.finishedTs)
|
||||||
|
list[idx] = old.copy(toolCall = oldJson.toString())
|
||||||
|
_messages.value = list
|
||||||
|
saveCurrentMessages()
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 创建/更新一条工具调用消息(按 toolCall JSON 的 id 匹配去重) */
|
||||||
|
private fun upsertToolCallMessage(call: ParsedToolCall) {
|
||||||
|
val list = _messages.value.toMutableList()
|
||||||
|
val existingIndex = list.indexOfFirst { msg ->
|
||||||
|
msg.toolCall?.let { toolCallIdOf(it) == call.id } == true
|
||||||
|
}
|
||||||
|
val normalized = buildToolCallJson(call)
|
||||||
|
if (existingIndex >= 0) {
|
||||||
|
list[existingIndex] = list[existingIndex].copy(toolCall = normalized)
|
||||||
|
} else {
|
||||||
|
list.add(
|
||||||
|
ChatMessage(
|
||||||
|
id = "tool_${call.id}",
|
||||||
|
content = "",
|
||||||
|
role = MessageRole.ASSISTANT,
|
||||||
|
sessionId = currentSessionId ?: "",
|
||||||
|
timestamp = call.ts ?: System.currentTimeMillis(),
|
||||||
|
streaming = false,
|
||||||
|
toolCall = normalized,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
_messages.value = list
|
||||||
|
saveCurrentMessages()
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 从分片文本中尽力提取工具调用 id(首个 `"id":"..."`),供分片路由使用 */
|
||||||
|
private fun extractToolCallId(chunk: String): String? {
|
||||||
|
val t = chunk.trimStart()
|
||||||
|
if (!t.startsWith("{")) return null
|
||||||
|
return TOOL_CALL_ID_REGEX.find(t)?.groupValues?.getOrNull(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 尝试把(可能积累后的)文本解析为工具调用;支持 AstrBot 紧凑格式与 OpenAI 格式;不完整返回 null */
|
||||||
|
private fun tryParseToolCall(raw: String): ParsedToolCall? {
|
||||||
|
val root = try {
|
||||||
|
JsonParser.parseString(raw)
|
||||||
|
} catch (e: Exception) {
|
||||||
|
return null
|
||||||
|
}
|
||||||
|
if (!root.isJsonObject) return null
|
||||||
|
val obj = root.asJsonObject
|
||||||
|
val id = obj.get("id")?.takeIf { it.isJsonPrimitive }?.asString?.takeIf { it.isNotEmpty() }
|
||||||
|
?: return null
|
||||||
|
|
||||||
|
var name: String? = null
|
||||||
|
var args: String? = null
|
||||||
|
var ts: Long? = null
|
||||||
|
|
||||||
|
// AstrBot 紧凑格式:{"id","name","args","ts"}
|
||||||
|
obj.get("name")?.takeIf { it.isJsonPrimitive }?.let { name = it.asString }
|
||||||
|
obj.get("args")?.let { args = jsonElementToText(it) }
|
||||||
|
obj.get("ts")?.takeIf { it.isJsonPrimitive && it.asJsonPrimitive.isNumber }?.let {
|
||||||
|
ts = (it.asDouble * 1000).toLong() // AstrBot 发的是 time.time() 秒,转毫秒
|
||||||
|
}
|
||||||
|
|
||||||
|
// OpenAI 格式:{"id","type":"function","function":{"name","arguments"}}
|
||||||
|
val fn = obj.get("function")?.takeIf { it.isJsonObject }?.asJsonObject
|
||||||
|
if (fn != null) {
|
||||||
|
if (name == null) {
|
||||||
|
fn.get("name")?.takeIf { it.isJsonPrimitive }?.let { name = it.asString }
|
||||||
|
}
|
||||||
|
if (args == null) {
|
||||||
|
fn.get("arguments")?.let { args = jsonElementToText(it) }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 至少识别出 name 或 args 才算有效工具调用(避免把纯 id 片段误当完整调用)
|
||||||
|
if (name == null && args == null) return null
|
||||||
|
return ParsedToolCall(
|
||||||
|
id = id,
|
||||||
|
name = name,
|
||||||
|
args = args,
|
||||||
|
result = null,
|
||||||
|
ts = ts ?: System.currentTimeMillis(),
|
||||||
|
finishedTs = null,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 解析工具调用结果 `{"id","result","ts"}`;不完整返回 null */
|
||||||
|
private fun tryParseToolCallResult(raw: String): ParsedToolCall? {
|
||||||
|
val root = try {
|
||||||
|
JsonParser.parseString(raw)
|
||||||
|
} catch (e: Exception) {
|
||||||
|
return null
|
||||||
|
}
|
||||||
|
if (!root.isJsonObject) return null
|
||||||
|
val obj = root.asJsonObject
|
||||||
|
val id = obj.get("id")?.takeIf { it.isJsonPrimitive }?.asString?.takeIf { it.isNotEmpty() }
|
||||||
|
?: return null
|
||||||
|
val result = obj.get("result")?.let { jsonElementToText(it) }
|
||||||
|
return ParsedToolCall(
|
||||||
|
id = id,
|
||||||
|
name = null,
|
||||||
|
args = null,
|
||||||
|
result = result,
|
||||||
|
ts = null,
|
||||||
|
finishedTs = System.currentTimeMillis(),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 从已存 toolCall JSON 中提取 id(用于结果回关联) */
|
||||||
|
private fun toolCallIdOf(toolCallJson: String): String? {
|
||||||
|
val obj = try {
|
||||||
|
JsonParser.parseString(toolCallJson)
|
||||||
|
} catch (e: Exception) {
|
||||||
|
return null
|
||||||
|
}
|
||||||
|
if (!obj.isJsonObject) return null
|
||||||
|
return obj.asJsonObject.get("id")?.takeIf { it.isJsonPrimitive }?.asString
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 构建工具调用消息的规范化 JSON(UI 契约):
|
||||||
|
* `{"id","name","args","result","ts","finished_ts"}`;args/result 均为字符串,UI 端负责 pretty-print。
|
||||||
|
*/
|
||||||
|
private fun buildToolCallJson(call: ParsedToolCall): String {
|
||||||
|
val obj = JsonObject()
|
||||||
|
obj.addProperty("id", call.id)
|
||||||
|
obj.addProperty("name", call.name ?: "")
|
||||||
|
obj.addProperty("args", call.args ?: "")
|
||||||
|
if (call.result != null) obj.addProperty("result", call.result)
|
||||||
|
obj.addProperty("ts", call.ts ?: System.currentTimeMillis())
|
||||||
|
obj.add("finished_ts", call.finishedTs?.let { JsonPrimitive(it) } ?: JsonNull.INSTANCE)
|
||||||
|
return obj.toString()
|
||||||
|
}
|
||||||
|
|
||||||
|
/** JsonElement → 字符串(对象/数组紧凑序列化,字符串原样返回),供 args/result 使用 */
|
||||||
|
private fun jsonElementToText(el: JsonElement): String = when {
|
||||||
|
el.isJsonNull -> ""
|
||||||
|
el.isJsonPrimitive && el.asJsonPrimitive.isString -> el.asString
|
||||||
|
else -> el.toString()
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun resetToolCallBuffers() {
|
||||||
|
toolCallBuffers.clear()
|
||||||
|
lastToolCallId = null
|
||||||
|
}
|
||||||
|
|
||||||
private fun saveCurrentMessages() {
|
private fun saveCurrentMessages() {
|
||||||
val sid = currentSessionId ?: return
|
val sid = currentSessionId ?: return
|
||||||
sessionMessages[sid] = _messages.value.toMutableList()
|
sessionMessages[sid] = _messages.value.toMutableList()
|
||||||
@@ -414,6 +690,7 @@ class ChatRepository(
|
|||||||
fun switchSession(sessionId: String) {
|
fun switchSession(sessionId: String) {
|
||||||
Log.d(TAG, "Switch to session $sessionId")
|
Log.d(TAG, "Switch to session $sessionId")
|
||||||
saveCurrentMessages()
|
saveCurrentMessages()
|
||||||
|
resetToolCallBuffers()
|
||||||
currentSessionId = sessionId
|
currentSessionId = sessionId
|
||||||
_messages.value = sessionMessages[sessionId]?.toList() ?: emptyList()
|
_messages.value = sessionMessages[sessionId]?.toList() ?: emptyList()
|
||||||
}
|
}
|
||||||
@@ -428,6 +705,7 @@ class ChatRepository(
|
|||||||
fun newSession() {
|
fun newSession() {
|
||||||
Log.d(TAG, "New session")
|
Log.d(TAG, "New session")
|
||||||
saveCurrentMessages()
|
saveCurrentMessages()
|
||||||
|
resetToolCallBuffers()
|
||||||
val placeholderId = "local_${UUID.randomUUID().toString().take(8)}"
|
val placeholderId = "local_${UUID.randomUUID().toString().take(8)}"
|
||||||
val emptySession = ChatSession(sessionId = placeholderId, displayName = "新会话")
|
val emptySession = ChatSession(sessionId = placeholderId, displayName = "新会话")
|
||||||
_sessions.value = _sessions.value + emptySession
|
_sessions.value = _sessions.value + emptySession
|
||||||
|
|||||||
@@ -41,11 +41,51 @@ class AppSettings(context: Context) {
|
|||||||
val isConfigured: Boolean
|
val isConfigured: Boolean
|
||||||
get() = apiKey.isNotBlank() && serverUrl.isNotBlank()
|
get() = apiKey.isNotBlank() && serverUrl.isNotBlank()
|
||||||
|
|
||||||
|
// ===== 语音通话(MiMo ASR/TTS)配置组 =====
|
||||||
|
|
||||||
|
var mimoApiKey: String
|
||||||
|
get() = prefs.getString(KEY_MIMO_API_KEY, "") ?: ""
|
||||||
|
set(value) = prefs.edit().putString(KEY_MIMO_API_KEY, value).apply()
|
||||||
|
|
||||||
|
var mimoBaseUrl: String
|
||||||
|
get() = prefs.getString(KEY_MIMO_BASE_URL, DEFAULT_MIMO_BASE_URL) ?: DEFAULT_MIMO_BASE_URL
|
||||||
|
set(value) = prefs.edit().putString(KEY_MIMO_BASE_URL, value).apply()
|
||||||
|
|
||||||
|
var asrModel: String
|
||||||
|
get() = prefs.getString(KEY_ASR_MODEL, DEFAULT_ASR_MODEL) ?: DEFAULT_ASR_MODEL
|
||||||
|
set(value) = prefs.edit().putString(KEY_ASR_MODEL, value).apply()
|
||||||
|
|
||||||
|
var ttsModel: String
|
||||||
|
get() = prefs.getString(KEY_TTS_MODEL, DEFAULT_TTS_MODEL) ?: DEFAULT_TTS_MODEL
|
||||||
|
set(value) = prefs.edit().putString(KEY_TTS_MODEL, value).apply()
|
||||||
|
|
||||||
|
var ttsVoice: String
|
||||||
|
get() = prefs.getString(KEY_TTS_VOICE, DEFAULT_TTS_VOICE) ?: DEFAULT_TTS_VOICE
|
||||||
|
set(value) = prefs.edit().putString(KEY_TTS_VOICE, value).apply()
|
||||||
|
|
||||||
|
var ttsFormat: String
|
||||||
|
get() = prefs.getString(KEY_TTS_FORMAT, DEFAULT_TTS_FORMAT) ?: DEFAULT_TTS_FORMAT
|
||||||
|
set(value) = prefs.edit().putString(KEY_TTS_FORMAT, value).apply()
|
||||||
|
|
||||||
|
val isVoiceConfigured: Boolean
|
||||||
|
get() = mimoApiKey.isNotBlank()
|
||||||
|
|
||||||
companion object {
|
companion object {
|
||||||
private const val KEY_SERVER_URL = "server_url"
|
private const val KEY_SERVER_URL = "server_url"
|
||||||
private const val KEY_API_KEY = "api_key"
|
private const val KEY_API_KEY = "api_key"
|
||||||
private const val KEY_USERNAME = "username"
|
private const val KEY_USERNAME = "username"
|
||||||
|
private const val KEY_MIMO_API_KEY = "mimo_api_key"
|
||||||
|
private const val KEY_MIMO_BASE_URL = "mimo_base_url"
|
||||||
|
private const val KEY_ASR_MODEL = "asr_model"
|
||||||
|
private const val KEY_TTS_MODEL = "tts_model"
|
||||||
|
private const val KEY_TTS_VOICE = "tts_voice"
|
||||||
|
private const val KEY_TTS_FORMAT = "tts_format"
|
||||||
private const val DEFAULT_SERVER_URL = "http://192.168.1.100:6185"
|
private const val DEFAULT_SERVER_URL = "http://192.168.1.100:6185"
|
||||||
private const val DEFAULT_USERNAME = "RainnyaUser"
|
private const val DEFAULT_USERNAME = "RainnyaUser"
|
||||||
|
private const val DEFAULT_MIMO_BASE_URL = "https://api.xiaomimimo.com/v1"
|
||||||
|
private const val DEFAULT_ASR_MODEL = "mimo-v2.5-asr"
|
||||||
|
private const val DEFAULT_TTS_MODEL = "mimo-v2.5-tts"
|
||||||
|
private const val DEFAULT_TTS_VOICE = "冰糖"
|
||||||
|
private const val DEFAULT_TTS_FORMAT = "pcm16"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,140 @@
|
|||||||
|
package com.rainnya.chat.data.upload
|
||||||
|
|
||||||
|
import android.content.Context
|
||||||
|
import android.net.Uri
|
||||||
|
import android.provider.OpenableColumns
|
||||||
|
import android.util.Log
|
||||||
|
import com.google.gson.Gson
|
||||||
|
import com.google.gson.JsonObject
|
||||||
|
import kotlinx.coroutines.Dispatchers
|
||||||
|
import kotlinx.coroutines.withContext
|
||||||
|
import okhttp3.MediaType
|
||||||
|
import okhttp3.MediaType.Companion.toMediaTypeOrNull
|
||||||
|
import okhttp3.MultipartBody
|
||||||
|
import okhttp3.OkHttpClient
|
||||||
|
import okhttp3.Request
|
||||||
|
import okhttp3.RequestBody
|
||||||
|
import okio.BufferedSink
|
||||||
|
import java.io.IOException
|
||||||
|
import java.util.concurrent.TimeUnit
|
||||||
|
|
||||||
|
private const val TAG = "RainnyaUploader"
|
||||||
|
|
||||||
|
class ImageUploader(
|
||||||
|
private val client: OkHttpClient = OkHttpClient.Builder()
|
||||||
|
.connectTimeout(15, TimeUnit.SECONDS)
|
||||||
|
.readTimeout(60, TimeUnit.SECONDS)
|
||||||
|
.writeTimeout(60, TimeUnit.SECONDS)
|
||||||
|
.build(),
|
||||||
|
) {
|
||||||
|
private val gson = Gson()
|
||||||
|
|
||||||
|
suspend fun upload(
|
||||||
|
context: Context,
|
||||||
|
baseUrl: String,
|
||||||
|
apiKey: String,
|
||||||
|
uri: Uri,
|
||||||
|
onProgress: (Float) -> Unit = {},
|
||||||
|
): String = withContext(Dispatchers.IO) {
|
||||||
|
try {
|
||||||
|
val fileName = queryDisplayName(context, uri)
|
||||||
|
?: "image_${System.currentTimeMillis()}.jpg"
|
||||||
|
val mimeType = context.contentResolver.getType(uri) ?: "image/jpeg"
|
||||||
|
val contentLength = queryContentLength(context, uri)
|
||||||
|
|
||||||
|
val inputStream = context.contentResolver.openInputStream(uri)
|
||||||
|
?: throw IOException("Cannot open input stream for uri: $uri")
|
||||||
|
|
||||||
|
val fileBody = object : RequestBody() {
|
||||||
|
override fun contentType(): MediaType? = mimeType.toMediaTypeOrNull()
|
||||||
|
|
||||||
|
override fun writeTo(sink: BufferedSink) {
|
||||||
|
var uploaded = 0L
|
||||||
|
val buffer = ByteArray(DEFAULT_BUFFER_SIZE)
|
||||||
|
inputStream.use { stream ->
|
||||||
|
while (true) {
|
||||||
|
val read = stream.read(buffer)
|
||||||
|
if (read == -1) break
|
||||||
|
sink.write(buffer, 0, read)
|
||||||
|
uploaded += read
|
||||||
|
if (contentLength > 0) {
|
||||||
|
onProgress(
|
||||||
|
(uploaded.toFloat() / contentLength.toFloat()).coerceIn(0f, 1f)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
val multipart = MultipartBody.Builder()
|
||||||
|
.setType(MultipartBody.FORM)
|
||||||
|
.addFormDataPart("file", fileName, fileBody)
|
||||||
|
.build()
|
||||||
|
|
||||||
|
val request = Request.Builder()
|
||||||
|
.url(baseUrl.trimEnd('/') + "/api/v1/file")
|
||||||
|
.header("Authorization", "Bearer $apiKey")
|
||||||
|
.post(multipart)
|
||||||
|
.build()
|
||||||
|
|
||||||
|
client.newCall(request).execute().use { response ->
|
||||||
|
val body = response.body?.string().orEmpty()
|
||||||
|
if (!response.isSuccessful) {
|
||||||
|
throw IOException("Upload failed: HTTP ${response.code} - $body")
|
||||||
|
}
|
||||||
|
val json = try {
|
||||||
|
gson.fromJson(body, JsonObject::class.java)
|
||||||
|
} catch (e: Exception) {
|
||||||
|
throw IOException("Invalid response: $body", e)
|
||||||
|
}
|
||||||
|
if (json.get("status")?.asString != "ok") {
|
||||||
|
val message = json.get("message")?.asString ?: "Unknown error"
|
||||||
|
throw IOException("Upload failed: $message")
|
||||||
|
}
|
||||||
|
val attachmentId = json.getAsJsonObject("data")?.get("attachment_id")?.asString
|
||||||
|
?: throw IOException("Missing attachment_id in response: $body")
|
||||||
|
onProgress(1f)
|
||||||
|
attachmentId
|
||||||
|
}
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.e(TAG, "Image upload failed: ${e.message}", e)
|
||||||
|
throw e
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun queryDisplayName(context: Context, uri: Uri): String? {
|
||||||
|
return try {
|
||||||
|
context.contentResolver.query(
|
||||||
|
uri,
|
||||||
|
arrayOf(OpenableColumns.DISPLAY_NAME),
|
||||||
|
null,
|
||||||
|
null,
|
||||||
|
null,
|
||||||
|
)?.use { cursor ->
|
||||||
|
if (cursor.moveToFirst()) {
|
||||||
|
val index = cursor.getColumnIndex(OpenableColumns.DISPLAY_NAME)
|
||||||
|
if (index >= 0) cursor.getString(index) else null
|
||||||
|
} else {
|
||||||
|
null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.w(TAG, "Failed to query display name", e)
|
||||||
|
null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun queryContentLength(context: Context, uri: Uri): Long {
|
||||||
|
return try {
|
||||||
|
context.contentResolver.openAssetFileDescriptor(uri, "r")?.length ?: -1L
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.w(TAG, "Failed to query content length", e)
|
||||||
|
-1L
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
companion object {
|
||||||
|
private const val DEFAULT_BUFFER_SIZE = 64 * 1024
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,216 @@
|
|||||||
|
package com.rainnya.chat.data.voice
|
||||||
|
|
||||||
|
import android.media.AudioFormat
|
||||||
|
import android.media.AudioTrack
|
||||||
|
import android.util.Log
|
||||||
|
import java.util.concurrent.LinkedBlockingQueue
|
||||||
|
import java.util.concurrent.TimeUnit
|
||||||
|
import java.util.concurrent.atomic.AtomicBoolean
|
||||||
|
import java.util.concurrent.atomic.AtomicLong
|
||||||
|
import kotlin.math.max
|
||||||
|
|
||||||
|
private const val TAG = "RainnyaAudioPlay"
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 语音播放引擎(M3):AudioTrack 播放 MiMo TTS 返回的 PCM 流。
|
||||||
|
*
|
||||||
|
* - AudioTrack 参数:24kHz / MONO / PCM_16BIT / MODE_STREAM,缓冲 = max(系统最小值, 500ms);
|
||||||
|
* - [enqueue] 带 seq 序号入队,内部按 seq 重排成连续块再进阻塞队列,防并行合成导致句子乱序(§3.6.3);
|
||||||
|
* - 后台写线程 + 阻塞队列 + `write(WRITE_BLOCKING)` 天然背压;
|
||||||
|
* - [play] 预滚 ~175ms 后再 play(),吞掉 SSE 抖动、避免开头断音;
|
||||||
|
* - 预滚耗尽时允许 1~2 次短暂静音(~50ms)兜底,不无限等待;
|
||||||
|
* - [stop] 只清队列 + pause + flush,为打断准备,**非销毁**;[release] 才释放线程与 AudioTrack。
|
||||||
|
* - 写线程为 daemon,随 [release] 退出;内网并发用队列 + 锁保证线程安全。
|
||||||
|
*/
|
||||||
|
class AudioPlaybackEngine(private val sampleRate: Int = 24_000) {
|
||||||
|
|
||||||
|
private val bytesPerSec = sampleRate * BYTES_PER_SAMPLE
|
||||||
|
|
||||||
|
/** AudioTrack 内部缓冲:max(系统最小值, 500ms 数据量),够吞 SSE 抖动 */
|
||||||
|
private val bufferSizeBytes = max(
|
||||||
|
AudioTrack.getMinBufferSize(
|
||||||
|
sampleRate,
|
||||||
|
AudioFormat.CHANNEL_OUT_MONO,
|
||||||
|
AudioFormat.ENCODING_PCM_16BIT,
|
||||||
|
),
|
||||||
|
bytesPerSec / 2,
|
||||||
|
)
|
||||||
|
|
||||||
|
/** 待播放 PCM 队列(已按 seq 重排成连续),写线程阻塞消费 */
|
||||||
|
private val queue = LinkedBlockingQueue<ByteArray>()
|
||||||
|
|
||||||
|
/** 乱序缓冲:seq -> PCM 块;[enqueue] 把从 nextSeq 起连续的块刷入 [queue] */
|
||||||
|
private val reorderLock = Any()
|
||||||
|
private val reorderBuffer = HashMap<Int, ByteArray>()
|
||||||
|
private var nextSeq = 0
|
||||||
|
|
||||||
|
@Volatile
|
||||||
|
private var track: AudioTrack? = null
|
||||||
|
|
||||||
|
/** 写线程是否已启动(防重复 start) */
|
||||||
|
private val started = AtomicBoolean(false)
|
||||||
|
|
||||||
|
/** 是否已释放(release 后拒绝任何操作) */
|
||||||
|
private val closed = AtomicBoolean(false)
|
||||||
|
private var writerThread: Thread? = null
|
||||||
|
|
||||||
|
/** 写线程累计写入 AudioTrack 的字节数(含静音块,供预滚/排空判断) */
|
||||||
|
private val writtenBytes = AtomicLong(0)
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 带 seq 入队。内部按 seq 重排:只把从 nextSeq 起连续编号的块送入播放队列,
|
||||||
|
* 乱序先到的块暂存在 [reorderBuffer],等缺的序号补上后再按序播放。
|
||||||
|
*/
|
||||||
|
fun enqueue(seq: Int, pcm: ByteArray) {
|
||||||
|
if (closed.get() || pcm.isEmpty()) return
|
||||||
|
val ready = ArrayList<ByteArray>()
|
||||||
|
synchronized(reorderLock) {
|
||||||
|
reorderBuffer[seq] = pcm
|
||||||
|
while (true) {
|
||||||
|
val chunk = reorderBuffer.remove(nextSeq) ?: break
|
||||||
|
ready.add(chunk)
|
||||||
|
nextSeq++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ready.forEach { queue.offer(it) }
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 预滚 ~175ms 后 play():吞 SSE 抖动,避免开头断音 */
|
||||||
|
fun play() {
|
||||||
|
if (closed.get()) return
|
||||||
|
ensureTrack()
|
||||||
|
startWriter()
|
||||||
|
val target = bytesPerSec * PRE_ROLL_MS / 1000
|
||||||
|
val deadline = System.currentTimeMillis() + PRE_ROLL_TIMEOUT_MS
|
||||||
|
while (!closed.get() && writtenBytes.get() < target &&
|
||||||
|
System.currentTimeMillis() < deadline
|
||||||
|
) {
|
||||||
|
Thread.sleep(10)
|
||||||
|
}
|
||||||
|
track?.let { if (it.playState != AudioTrack.PLAYSTATE_PLAYING) it.play() }
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 打断准备(非销毁):清空待播队列 + 暂停 + 清空 AudioTrack 缓冲 */
|
||||||
|
fun stop() {
|
||||||
|
queue.clear()
|
||||||
|
synchronized(reorderLock) {
|
||||||
|
reorderBuffer.clear()
|
||||||
|
nextSeq = 0
|
||||||
|
}
|
||||||
|
writtenBytes.set(0)
|
||||||
|
track?.let { t ->
|
||||||
|
try {
|
||||||
|
if (t.playState == AudioTrack.PLAYSTATE_PLAYING) t.pause()
|
||||||
|
t.flush()
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.w(TAG, "stop 失败", e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 释放写线程与 AudioTrack(随引擎 scope 取消调用) */
|
||||||
|
fun release() {
|
||||||
|
if (!closed.compareAndSet(false, true)) return
|
||||||
|
queue.clear()
|
||||||
|
synchronized(reorderLock) { reorderBuffer.clear() }
|
||||||
|
track?.let { t ->
|
||||||
|
try {
|
||||||
|
if (t.playState == AudioTrack.PLAYSTATE_PLAYING) t.stop()
|
||||||
|
t.release()
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.w(TAG, "release 失败", e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
track = null
|
||||||
|
writerThread?.let { t ->
|
||||||
|
if (t.isAlive) t.interrupt()
|
||||||
|
try {
|
||||||
|
t.join(1000)
|
||||||
|
} catch (e: InterruptedException) {
|
||||||
|
Thread.currentThread().interrupt()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
writerThread = null
|
||||||
|
started.set(false)
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 已写入 AudioTrack 的字节数(含静音块) */
|
||||||
|
fun writtenBytesCount(): Long = writtenBytes.get()
|
||||||
|
|
||||||
|
/** AudioTrack 已播放的字节数(playbackHeadPosition × 2B);未建 track 视为全播完 */
|
||||||
|
fun playedBytesCount(): Long {
|
||||||
|
val t = track ?: return writtenBytes.get()
|
||||||
|
return t.playbackHeadPosition.toLong() * BYTES_PER_SAMPLE
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun ensureTrack() {
|
||||||
|
if (track != null) return
|
||||||
|
track = try {
|
||||||
|
AudioTrack.Builder()
|
||||||
|
.setAudioFormat(
|
||||||
|
AudioFormat.Builder()
|
||||||
|
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
|
||||||
|
.setSampleRate(sampleRate)
|
||||||
|
.setChannelMask(AudioFormat.CHANNEL_OUT_MONO)
|
||||||
|
.build(),
|
||||||
|
)
|
||||||
|
.setTransferMode(AudioTrack.MODE_STREAM)
|
||||||
|
.setBufferSizeInBytes(bufferSizeBytes)
|
||||||
|
.build()
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.e(TAG, "创建 AudioTrack 失败", e)
|
||||||
|
null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun startWriter() {
|
||||||
|
if (!started.compareAndSet(false, true)) return
|
||||||
|
val audio = track ?: run {
|
||||||
|
started.set(false)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
val t = Thread({
|
||||||
|
try {
|
||||||
|
var silenceBursts = 0
|
||||||
|
while (!closed.get()) {
|
||||||
|
val chunk = queue.poll(POLL_MS, TimeUnit.MILLISECONDS)
|
||||||
|
if (chunk == null) {
|
||||||
|
// 预滚耗尽:允许 1~2 次短暂静音兜底,不无限等(抗卡顿)
|
||||||
|
if (silenceBursts < MAX_SILENCE_BURSTS && writtenBytes.get() > 0) {
|
||||||
|
silenceBursts++
|
||||||
|
val silence = ByteArray(bytesPerSec * SILENCE_MS / 1000)
|
||||||
|
try {
|
||||||
|
audio.write(silence, 0, silence.size, AudioTrack.WRITE_BLOCKING)
|
||||||
|
writtenBytes.addAndGet(silence.size.toLong())
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.w(TAG, "写静音失败", e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
silenceBursts = 0
|
||||||
|
try {
|
||||||
|
audio.write(chunk, 0, chunk.size, AudioTrack.WRITE_BLOCKING)
|
||||||
|
writtenBytes.addAndGet(chunk.size.toLong())
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.e(TAG, "写入 AudioTrack 失败", e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
started.set(false)
|
||||||
|
}
|
||||||
|
}, "RainnyaPlayback")
|
||||||
|
writerThread = t
|
||||||
|
t.isDaemon = true
|
||||||
|
t.start()
|
||||||
|
}
|
||||||
|
|
||||||
|
companion object {
|
||||||
|
private const val BYTES_PER_SAMPLE = 2
|
||||||
|
private const val PRE_ROLL_MS = 175
|
||||||
|
private const val PRE_ROLL_TIMEOUT_MS = 3_000
|
||||||
|
private const val SILENCE_MS = 50
|
||||||
|
private const val MAX_SILENCE_BURSTS = 2
|
||||||
|
private const val POLL_MS = 100L
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,286 @@
|
|||||||
|
package com.rainnya.chat.data.voice
|
||||||
|
|
||||||
|
import android.media.AudioFormat
|
||||||
|
import android.media.AudioRecord
|
||||||
|
import android.media.MediaRecorder
|
||||||
|
import android.util.Log
|
||||||
|
import kotlinx.coroutines.Dispatchers
|
||||||
|
import kotlinx.coroutines.ensureActive
|
||||||
|
import kotlinx.coroutines.withContext
|
||||||
|
import java.io.ByteArrayOutputStream
|
||||||
|
import java.io.IOException
|
||||||
|
import java.nio.ByteBuffer
|
||||||
|
import java.nio.ByteOrder
|
||||||
|
import kotlin.math.max
|
||||||
|
import kotlin.math.sqrt
|
||||||
|
|
||||||
|
private const val TAG = "RainnyaAudioRec"
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 录音 + VAD(M0)。
|
||||||
|
*
|
||||||
|
* 内部使用 AudioRecord 以 16kHz / 16bit / 单声道 采集,返回带 RIFF 头的 WAV 字节数组。
|
||||||
|
* VAD 策略(v1):
|
||||||
|
* - 能量用 RMS(短时均方根)衡量;
|
||||||
|
* - 阈值 = max(固定下限, 前 500ms 噪声均值 × 系数),避免动态无下限导致耳语永不达标;
|
||||||
|
* - 检测到语音后持续静音 600ms 自动结束;
|
||||||
|
* - 45s 硬切防成本失控;
|
||||||
|
* - 45s 内全程未超过阈值("未检测到语音")→ 返回 null,由上层丢弃、不发 ASR。
|
||||||
|
*
|
||||||
|
* 权限被拒(SecurityException)/ 设备被占用(IllegalStateException)→ 向上抛异常,
|
||||||
|
* 由 VoiceCallEngine 映射为中文错误提示。
|
||||||
|
*/
|
||||||
|
class AudioRecorder {
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 带 VAD 的录音,返回 WAV(16k/16bit/mono,含 RIFF 头);
|
||||||
|
* 全程未检测到语音时返回 null。
|
||||||
|
*/
|
||||||
|
suspend fun recordWithVad(): ByteArray? = withContext(Dispatchers.IO) {
|
||||||
|
val minBuffer = AudioRecord.getMinBufferSize(
|
||||||
|
SAMPLE_RATE,
|
||||||
|
AudioFormat.CHANNEL_IN_MONO,
|
||||||
|
AudioFormat.ENCODING_PCM_16BIT,
|
||||||
|
)
|
||||||
|
val bufferBytes = max(minBuffer, SAMPLE_RATE * BYTES_PER_SAMPLE / 10) // 至少 100ms 缓冲(16000Hz×2B×0.1s=3200B;此前误写 ×100 分配 3.2MB 浪费内存)
|
||||||
|
|
||||||
|
val record: AudioRecord = try {
|
||||||
|
AudioRecord.Builder()
|
||||||
|
.setAudioSource(MediaRecorder.AudioSource.MIC)
|
||||||
|
.setAudioFormat(
|
||||||
|
AudioFormat.Builder()
|
||||||
|
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
|
||||||
|
.setSampleRate(SAMPLE_RATE)
|
||||||
|
.setChannelMask(AudioFormat.CHANNEL_IN_MONO)
|
||||||
|
.build(),
|
||||||
|
)
|
||||||
|
.setBufferSizeInBytes(bufferBytes)
|
||||||
|
.build()
|
||||||
|
} catch (e: SecurityException) {
|
||||||
|
throw e
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
if (record.state != AudioRecord.STATE_INITIALIZED) {
|
||||||
|
throw IllegalStateException("录音设备不可用或被占用")
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
record.startRecording()
|
||||||
|
} catch (e: SecurityException) {
|
||||||
|
throw e
|
||||||
|
}
|
||||||
|
if (record.recordingState != AudioRecord.RECORDSTATE_RECORDING) {
|
||||||
|
throw IllegalStateException("录音设备被占用")
|
||||||
|
}
|
||||||
|
|
||||||
|
val pcm = ByteArrayOutputStream()
|
||||||
|
val chunk = ShortArray(CHUNK_SAMPLES)
|
||||||
|
val startMs = System.currentTimeMillis()
|
||||||
|
|
||||||
|
// 前 500ms 噪声校准:累积未检测到语音阶段的 RMS 求均值
|
||||||
|
var calibrationSum = 0.0
|
||||||
|
var calibrationCount = 0
|
||||||
|
var thresholdReady = false
|
||||||
|
var threshold = FIXED_FLOOR
|
||||||
|
|
||||||
|
var voiceDetected = false
|
||||||
|
var silenceMs = 0L
|
||||||
|
var endedBySilence = false
|
||||||
|
|
||||||
|
while (true) {
|
||||||
|
ensureActive() // 协程取消时快速退出,供 stopTalking/hangUp 打断
|
||||||
|
val elapsed = System.currentTimeMillis() - startMs
|
||||||
|
if (elapsed >= MAX_RECORD_MS) break // 45s 硬切
|
||||||
|
|
||||||
|
val read = record.read(chunk, 0, chunk.size, AudioRecord.READ_BLOCKING)
|
||||||
|
if (read < 0) {
|
||||||
|
throw IOException("录音读取失败,code=$read")
|
||||||
|
}
|
||||||
|
if (read == 0) continue
|
||||||
|
|
||||||
|
// 短整型 → 小端字节,写入 PCM 流
|
||||||
|
val bytes = ByteArray(read * BYTES_PER_SAMPLE)
|
||||||
|
ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN)
|
||||||
|
.asShortBuffer().put(chunk, 0, read)
|
||||||
|
pcm.write(bytes, 0, bytes.size)
|
||||||
|
|
||||||
|
val rms = rms(chunk, read)
|
||||||
|
|
||||||
|
if (!thresholdReady) {
|
||||||
|
// 只累计明显低于语音能量的块(RMS < FIXED_FLOOR)进噪声均值,
|
||||||
|
// 避免"点击即开口"把说话块混入校准,导致阈值被抬高、秒说话被误判为没听清
|
||||||
|
if (rms < FIXED_FLOOR) {
|
||||||
|
calibrationSum += rms
|
||||||
|
calibrationCount++
|
||||||
|
}
|
||||||
|
if (elapsed >= CALIBRATION_MS) {
|
||||||
|
val noiseMean = if (calibrationCount > 0) {
|
||||||
|
calibrationSum / calibrationCount
|
||||||
|
} else {
|
||||||
|
0.0
|
||||||
|
}
|
||||||
|
threshold = max(FIXED_FLOOR, noiseMean * NOISE_COEF)
|
||||||
|
thresholdReady = true
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if (rms >= threshold) {
|
||||||
|
voiceDetected = true
|
||||||
|
silenceMs = 0L
|
||||||
|
} else if (voiceDetected) {
|
||||||
|
silenceMs += CHUNK_DURATION_MS
|
||||||
|
if (silenceMs >= SILENCE_MS) {
|
||||||
|
endedBySilence = true
|
||||||
|
break // 600ms 持续静音 → 结束
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!voiceDetected) {
|
||||||
|
Log.w(TAG, "No voice detected within ${MAX_RECORD_MS}ms, discard recording")
|
||||||
|
return@withContext null
|
||||||
|
}
|
||||||
|
|
||||||
|
val pcmBytes = pcm.toByteArray()
|
||||||
|
Log.d(TAG, "Recorded ${pcmBytes.size} bytes (${pcmBytes.size / BYTES_PER_SAMPLE / 16}ms), " +
|
||||||
|
"silenceEnded=$endedBySilence")
|
||||||
|
buildWav(pcmBytes)
|
||||||
|
} finally {
|
||||||
|
try {
|
||||||
|
record.stop()
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.w(TAG, "Failed to stop AudioRecord", e)
|
||||||
|
}
|
||||||
|
record.release()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Barge-in 打断监听(M4):播放雨喵语音期间轻量监听麦克风,
|
||||||
|
* 只算 RMS 不存数据,不自动结束、不返回音频。
|
||||||
|
*
|
||||||
|
* - 连续读 AudioRecord 缓冲(20ms 一块),RMS 持续 ≥300ms 超过 [threshold] 即返回(触发打断信号);
|
||||||
|
* - 阈值由调用方(VoiceCallEngine)传入,播放态需上调(回声对策,§6/P1);
|
||||||
|
* - 权限被拒(SecurityException)/ 设备被占用(IllegalStateException)→ 向上抛异常,
|
||||||
|
* 由 VoiceCallEngine 决定是否降级;
|
||||||
|
* - 协程取消(CancellationException)→ 重抛,供引擎离开播放态 / 挂断时退出监听。
|
||||||
|
*/
|
||||||
|
suspend fun listenForBargeIn(threshold: Float): Unit = withContext(Dispatchers.IO) {
|
||||||
|
val minBuffer = AudioRecord.getMinBufferSize(
|
||||||
|
SAMPLE_RATE,
|
||||||
|
AudioFormat.CHANNEL_IN_MONO,
|
||||||
|
AudioFormat.ENCODING_PCM_16BIT,
|
||||||
|
)
|
||||||
|
val bufferBytes = max(minBuffer, SAMPLE_RATE * BYTES_PER_SAMPLE / 10) // 至少 100ms 缓冲
|
||||||
|
|
||||||
|
val record: AudioRecord = try {
|
||||||
|
AudioRecord.Builder()
|
||||||
|
.setAudioSource(MediaRecorder.AudioSource.MIC)
|
||||||
|
.setAudioFormat(
|
||||||
|
AudioFormat.Builder()
|
||||||
|
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
|
||||||
|
.setSampleRate(SAMPLE_RATE)
|
||||||
|
.setChannelMask(AudioFormat.CHANNEL_IN_MONO)
|
||||||
|
.build(),
|
||||||
|
)
|
||||||
|
.setBufferSizeInBytes(bufferBytes)
|
||||||
|
.build()
|
||||||
|
} catch (e: SecurityException) {
|
||||||
|
throw e
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
if (record.state != AudioRecord.STATE_INITIALIZED) {
|
||||||
|
throw IllegalStateException("录音设备不可用或被占用")
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
record.startRecording()
|
||||||
|
} catch (e: SecurityException) {
|
||||||
|
throw e
|
||||||
|
}
|
||||||
|
if (record.recordingState != AudioRecord.RECORDSTATE_RECORDING) {
|
||||||
|
throw IllegalStateException("录音设备被占用")
|
||||||
|
}
|
||||||
|
|
||||||
|
val chunk = ShortArray(CHUNK_SAMPLES)
|
||||||
|
var overMs = 0L
|
||||||
|
while (true) {
|
||||||
|
ensureActive() // 离开播放态/挂断时快速退出(READ_BLOCKING 单块最迟 ~20ms)
|
||||||
|
val read = record.read(chunk, 0, chunk.size, AudioRecord.READ_BLOCKING)
|
||||||
|
if (read < 0) {
|
||||||
|
throw IOException("录音读取失败,code=$read")
|
||||||
|
}
|
||||||
|
if (read == 0) continue
|
||||||
|
ensureActive() // 读取期间可能被取消,提前检查减少竞态
|
||||||
|
val rms = rms(chunk, read)
|
||||||
|
if (rms >= threshold) {
|
||||||
|
overMs += CHUNK_DURATION_MS
|
||||||
|
if (overMs >= BARGE_IN_HOLD_MS) {
|
||||||
|
return@withContext // 持续 ≥300ms 超阈值 → 打断信号
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
overMs = 0L // 播放回声的瞬时峰值不算(持续判定,§6/P1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
try {
|
||||||
|
record.stop()
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.w(TAG, "Failed to stop AudioRecord", e)
|
||||||
|
}
|
||||||
|
record.release()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 短时 RMS 能量,范围 0..32767 */
|
||||||
|
private fun rms(shortData: ShortArray, count: Int): Double {
|
||||||
|
if (count <= 0) return 0.0
|
||||||
|
var sum = 0.0
|
||||||
|
for (i in 0 until count) {
|
||||||
|
val v = shortData[i].toInt()
|
||||||
|
sum += v.toDouble() * v
|
||||||
|
}
|
||||||
|
return sqrt(sum / count)
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 拼接 44 字节 RIFF/WAVE 头 + PCM 数据 */
|
||||||
|
private fun buildWav(pcm: ByteArray): ByteArray {
|
||||||
|
val dataSize = pcm.size
|
||||||
|
val headerSize = 44
|
||||||
|
val wav = ByteArray(headerSize + dataSize)
|
||||||
|
val bb = ByteBuffer.wrap(wav).order(ByteOrder.LITTLE_ENDIAN)
|
||||||
|
|
||||||
|
bb.put("RIFF".toByteArray(Charsets.US_ASCII))
|
||||||
|
bb.putInt(36 + dataSize)
|
||||||
|
bb.put("WAVE".toByteArray(Charsets.US_ASCII))
|
||||||
|
bb.put("fmt ".toByteArray(Charsets.US_ASCII))
|
||||||
|
bb.putInt(16) // fmt chunk 大小
|
||||||
|
bb.putShort(1.toShort()) // PCM 编码
|
||||||
|
bb.putShort(1.toShort()) // 单声道
|
||||||
|
bb.putInt(SAMPLE_RATE) // 采样率 16k
|
||||||
|
bb.putInt(SAMPLE_RATE * BYTES_PER_SAMPLE) // 字节率 = 采样率 × 字节/样本
|
||||||
|
bb.putShort(BYTES_PER_SAMPLE.toShort()) // 块对齐
|
||||||
|
bb.putShort(BITS_PER_SAMPLE.toShort()) // 位深 16
|
||||||
|
bb.put("data".toByteArray(Charsets.US_ASCII))
|
||||||
|
bb.putInt(dataSize)
|
||||||
|
|
||||||
|
System.arraycopy(pcm, 0, wav, headerSize, dataSize)
|
||||||
|
return wav
|
||||||
|
}
|
||||||
|
|
||||||
|
companion object {
|
||||||
|
private const val SAMPLE_RATE = 16_000
|
||||||
|
private const val BYTES_PER_SAMPLE = 2
|
||||||
|
private const val BITS_PER_SAMPLE = 16
|
||||||
|
private const val CHUNK_DURATION_MS = 20
|
||||||
|
private const val CHUNK_SAMPLES = SAMPLE_RATE * CHUNK_DURATION_MS / 1000
|
||||||
|
private const val CALIBRATION_MS = 500L
|
||||||
|
private const val SILENCE_MS = 600L
|
||||||
|
private const val MAX_RECORD_MS = 45_000L
|
||||||
|
private const val FIXED_FLOOR = 800.0
|
||||||
|
private const val NOISE_COEF = 2.0
|
||||||
|
|
||||||
|
/** barge-in:RMS 持续超过阈值达到该时长才判定开口(回声对策,§6/P1:300ms 持续判定) */
|
||||||
|
private const val BARGE_IN_HOLD_MS = 300L
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,168 @@
|
|||||||
|
package com.rainnya.chat.data.voice
|
||||||
|
|
||||||
|
import android.util.Log
|
||||||
|
import com.google.gson.Gson
|
||||||
|
import com.google.gson.JsonArray
|
||||||
|
import com.google.gson.JsonObject
|
||||||
|
import kotlinx.coroutines.CancellationException
|
||||||
|
import kotlinx.coroutines.Dispatchers
|
||||||
|
import kotlinx.coroutines.withContext
|
||||||
|
import okhttp3.Call
|
||||||
|
import okhttp3.MediaType.Companion.toMediaType
|
||||||
|
import okhttp3.OkHttpClient
|
||||||
|
import okhttp3.Request
|
||||||
|
import okhttp3.RequestBody.Companion.toRequestBody
|
||||||
|
import java.io.IOException
|
||||||
|
import java.util.Base64
|
||||||
|
import java.util.concurrent.TimeUnit
|
||||||
|
|
||||||
|
private const val TAG = "RainnyaAsr"
|
||||||
|
|
||||||
|
/**
|
||||||
|
* MiMo ASR 客户端(M1)。
|
||||||
|
*
|
||||||
|
* 通过 OpenAI 兼容端点 POST {baseUrl}/chat/completions 上传 WAV(base64) 完成识别。
|
||||||
|
* 构造时会规范化 baseUrl:不以 `/v1` 结尾则自动补 `/v1`(用户填 `https://api.xiaomimimo.com`
|
||||||
|
* 这类根地址也能直接工作),再拼接 /chat/completions。
|
||||||
|
*
|
||||||
|
* 鉴权头:Authorization: Bearer <apiKey>
|
||||||
|
* 返回 choices[0].message.content(string 或 [{type:text,text:…}] 数组);
|
||||||
|
* HTTP 非 2xx / status 非 ok → 抛 IOException。
|
||||||
|
*
|
||||||
|
* OkHttpClient 为伴生对象共享的单例(L1:避免每次 ASR 新建连接池);
|
||||||
|
* [cancel] 可中止 in-flight 请求(M2:供引擎在 stopTalking/hangUp 时调用)。
|
||||||
|
*/
|
||||||
|
class MiMoAsrClient(
|
||||||
|
baseUrl: String,
|
||||||
|
private val apiKey: String,
|
||||||
|
) {
|
||||||
|
/** ASR 模型名,由 VoiceCallEngine 注入 settings.asrModel */
|
||||||
|
var model: String = DEFAULT_MODEL
|
||||||
|
|
||||||
|
/** 规范化后的 baseUrl(保证以 /v1 结尾) */
|
||||||
|
private val normalizedBaseUrl = normalizeBaseUrl(baseUrl)
|
||||||
|
|
||||||
|
private val gson = Gson()
|
||||||
|
|
||||||
|
/** 当前 in-flight 请求的 Call,供 [cancel] 中止 */
|
||||||
|
@Volatile
|
||||||
|
private var activeCall: Call? = null
|
||||||
|
|
||||||
|
/** 中止当前 ASR HTTP 请求(阻塞中的 execute() 会被打断并抛出 IOException) */
|
||||||
|
fun cancel() {
|
||||||
|
activeCall?.cancel()
|
||||||
|
activeCall = null
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 识别 WAV(16k/16bit/mono,含 RIFF 头),返回识别文本;失败抛 IOException */
|
||||||
|
suspend fun transcribe(wav: ByteArray): String = withContext(Dispatchers.IO) {
|
||||||
|
if (wav.isEmpty()) throw IOException("音频数据为空")
|
||||||
|
val base64Audio = Base64.getEncoder().encodeToString(wav)
|
||||||
|
if (base64Audio.length > MAX_BASE64_LENGTH) {
|
||||||
|
throw IOException("音频过大,超过 10MB 编码上限")
|
||||||
|
}
|
||||||
|
|
||||||
|
val body = buildRequestBody(base64Audio)
|
||||||
|
val request = Request.Builder()
|
||||||
|
.url(normalizedBaseUrl + "/chat/completions")
|
||||||
|
.header("Authorization", "Bearer $apiKey")
|
||||||
|
.post(body.toString().toRequestBody(JSON_MEDIA_TYPE))
|
||||||
|
.build()
|
||||||
|
|
||||||
|
val call = sharedClient.newCall(request)
|
||||||
|
activeCall = call
|
||||||
|
try {
|
||||||
|
call.execute().use { response ->
|
||||||
|
val responseBody = response.body?.string().orEmpty()
|
||||||
|
if (!response.isSuccessful) {
|
||||||
|
throw IOException("ASR 请求失败:HTTP ${response.code} - $responseBody")
|
||||||
|
}
|
||||||
|
val json = try {
|
||||||
|
gson.fromJson(responseBody, JsonObject::class.java)
|
||||||
|
} catch (e: Exception) {
|
||||||
|
throw IOException("ASR 响应解析失败:$responseBody", e)
|
||||||
|
}
|
||||||
|
// status 显式存在且非 ok → 失败
|
||||||
|
json.get("status")?.takeIf { it.isJsonPrimitive }?.let { st ->
|
||||||
|
if (st.asString != "ok") {
|
||||||
|
throw IOException("ASR 状态异常:${st.asString}")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// 兼容 choices 在顶层 或 在 data.choices 两种结构
|
||||||
|
val topChoices = json.get("choices")?.takeIf { it.isJsonArray }?.asJsonArray
|
||||||
|
val dataChoices = json.get("data")?.takeIf { it.isJsonObject }?.asJsonObject
|
||||||
|
?.get("choices")?.takeIf { it.isJsonArray }?.asJsonArray
|
||||||
|
val choices: JsonArray? = topChoices ?: dataChoices
|
||||||
|
val content = choices
|
||||||
|
?.firstOrNull()?.asJsonObject
|
||||||
|
?.getAsJsonObject("message")
|
||||||
|
?.get("content")
|
||||||
|
val contentText: String? = when {
|
||||||
|
content == null -> null
|
||||||
|
content.isJsonPrimitive -> content.asString
|
||||||
|
// content 为数组形式([{type:text,text:…}])→ 取首个 text 元素兜底
|
||||||
|
content.isJsonArray -> content.asJsonArray
|
||||||
|
.firstOrNull()?.asJsonObject
|
||||||
|
?.get("text")
|
||||||
|
?.takeIf { it.isJsonPrimitive }
|
||||||
|
?.asString
|
||||||
|
else -> null
|
||||||
|
}
|
||||||
|
if (contentText == null) {
|
||||||
|
throw IOException("ASR 响应缺少识别文本:$responseBody")
|
||||||
|
}
|
||||||
|
Log.v(TAG, "Transcribed: $contentText")
|
||||||
|
contentText
|
||||||
|
}
|
||||||
|
} catch (e: IOException) {
|
||||||
|
// 被 [cancel] 中止 → 转成协程取消信号,避免被误判为网络错误
|
||||||
|
if (call.isCanceled()) throw CancellationException("ASR request cancelled")
|
||||||
|
throw e
|
||||||
|
} finally {
|
||||||
|
if (activeCall === call) activeCall = null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun buildRequestBody(base64Audio: String): JsonObject = JsonObject().apply {
|
||||||
|
addProperty("model", model)
|
||||||
|
val messages = JsonArray()
|
||||||
|
val userMsg = JsonObject().apply {
|
||||||
|
addProperty("role", "user")
|
||||||
|
val content = JsonArray()
|
||||||
|
val inputAudio = JsonObject().apply {
|
||||||
|
addProperty("data", "data:audio/wav;base64,$base64Audio")
|
||||||
|
}
|
||||||
|
val contentItem = JsonObject().apply {
|
||||||
|
addProperty("type", "input_audio")
|
||||||
|
add("input_audio", inputAudio)
|
||||||
|
}
|
||||||
|
content.add(contentItem)
|
||||||
|
add("content", content)
|
||||||
|
}
|
||||||
|
messages.add(userMsg)
|
||||||
|
add("messages", messages)
|
||||||
|
val asrOptions = JsonObject().apply {
|
||||||
|
addProperty("language", "zh")
|
||||||
|
}
|
||||||
|
add("asr_options", asrOptions)
|
||||||
|
}
|
||||||
|
|
||||||
|
companion object {
|
||||||
|
private val JSON_MEDIA_TYPE = "application/json; charset=utf-8".toMediaType()
|
||||||
|
private const val DEFAULT_MODEL = "mimo-v2.5-asr"
|
||||||
|
private const val MAX_BASE64_LENGTH = 10 * 1024 * 1024
|
||||||
|
|
||||||
|
/** 全实例共享的连接池/客户端,避免每次 ASR 新建(L1) */
|
||||||
|
private val sharedClient = OkHttpClient.Builder()
|
||||||
|
.connectTimeout(10, TimeUnit.SECONDS)
|
||||||
|
.readTimeout(30, TimeUnit.SECONDS)
|
||||||
|
.writeTimeout(30, TimeUnit.SECONDS)
|
||||||
|
.build()
|
||||||
|
|
||||||
|
/** 规范化 baseUrl:去尾部斜杠,不以 /v1 结尾时补 /v1(M5) */
|
||||||
|
private fun normalizeBaseUrl(raw: String): String {
|
||||||
|
val trimmed = raw.trimEnd('/')
|
||||||
|
return if (trimmed.endsWith("/v1")) trimmed else "$trimmed/v1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,171 @@
|
|||||||
|
package com.rainnya.chat.data.voice
|
||||||
|
|
||||||
|
import android.util.Log
|
||||||
|
import com.google.gson.Gson
|
||||||
|
import com.google.gson.JsonArray
|
||||||
|
import com.google.gson.JsonObject
|
||||||
|
import kotlinx.coroutines.CancellationException
|
||||||
|
import kotlinx.coroutines.Dispatchers
|
||||||
|
import kotlinx.coroutines.flow.Flow
|
||||||
|
import kotlinx.coroutines.flow.flow
|
||||||
|
import kotlinx.coroutines.withContext
|
||||||
|
import okhttp3.Call
|
||||||
|
import okhttp3.MediaType.Companion.toMediaType
|
||||||
|
import okhttp3.OkHttpClient
|
||||||
|
import okhttp3.Request
|
||||||
|
import okhttp3.RequestBody.Companion.toRequestBody
|
||||||
|
import java.io.IOException
|
||||||
|
import java.util.Base64
|
||||||
|
import java.util.concurrent.TimeUnit
|
||||||
|
|
||||||
|
private const val TAG = "RainnyaTts"
|
||||||
|
|
||||||
|
/**
|
||||||
|
* MiMo TTS 客户端(M3)。
|
||||||
|
*
|
||||||
|
* 通过 OpenAI 兼容端点 `POST {baseUrl}/chat/completions` 流式合成语音:
|
||||||
|
* - model = settings.ttsModel(默认 mimo-v2.5-tts);
|
||||||
|
* - 消息结构硬性要求:user 放风格指令,**要合成的文字必须放 assistant**;
|
||||||
|
* - `audio: {format:"pcm16", voice:...}` + `stream:true` → SSE 分块返回;
|
||||||
|
* - 每块 `choices[0].delta.audio.data` 为 Base64 编码的 **24kHz PCM16LE 单声道**;
|
||||||
|
* - 鉴权头 `Authorization: Bearer <apiKey>`。
|
||||||
|
*
|
||||||
|
* OkHttp readTimeout(0):长句合成可能 60s+ 无数据,防中途断流;
|
||||||
|
* [cancel] 可中止 in-flight SSE(打断原子操作之一,M3);主动 cancel 把 IOException
|
||||||
|
* 转成 CancellationException(对齐 MiMoAsrClient 的 M2 做法)。
|
||||||
|
*/
|
||||||
|
class MiMoTtsClient(
|
||||||
|
baseUrl: String,
|
||||||
|
private val apiKey: String,
|
||||||
|
private val model: String,
|
||||||
|
private val voice: String,
|
||||||
|
) {
|
||||||
|
/** 规范化后的 baseUrl(保证以 /v1 结尾) */
|
||||||
|
private val normalizedBaseUrl = normalizeBaseUrl(baseUrl)
|
||||||
|
|
||||||
|
private val gson = Gson()
|
||||||
|
|
||||||
|
/** 当前 in-flight SSE 请求的 Call,供 [cancel] 中止 */
|
||||||
|
@Volatile
|
||||||
|
private var activeCall: Call? = null
|
||||||
|
|
||||||
|
/** 中止当前 TTS SSE 请求(阻塞中的 readUtf8Line 会抛 IOException) */
|
||||||
|
fun cancel() {
|
||||||
|
activeCall?.cancel()
|
||||||
|
activeCall = null
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 合成单句文本,流式返回 PCM 块(24kHz/16bit/mono)。
|
||||||
|
* 被 [cancel] 中止时抛 CancellationException,由调用方按打断处理。
|
||||||
|
*/
|
||||||
|
fun synthesizeStream(text: String): Flow<ByteArray> = flow {
|
||||||
|
if (text.isBlank()) return@flow
|
||||||
|
val request = Request.Builder()
|
||||||
|
.url(normalizedBaseUrl + "/chat/completions")
|
||||||
|
.header("Authorization", "Bearer $apiKey")
|
||||||
|
.post(buildRequestBody(text).toString().toRequestBody(JSON_MEDIA_TYPE))
|
||||||
|
.build()
|
||||||
|
|
||||||
|
val call = sharedClient.newCall(request)
|
||||||
|
activeCall = call
|
||||||
|
try {
|
||||||
|
withContext(Dispatchers.IO) {
|
||||||
|
call.execute().use { response ->
|
||||||
|
if (!response.isSuccessful) {
|
||||||
|
throw IOException("TTS 请求失败:HTTP ${response.code}")
|
||||||
|
}
|
||||||
|
val source = response.body?.source()
|
||||||
|
?: throw IOException("TTS 响应无内容")
|
||||||
|
while (true) {
|
||||||
|
val line = source.readUtf8Line() ?: break
|
||||||
|
val audioData = parseSseLine(line)
|
||||||
|
if (audioData == DONE) break
|
||||||
|
if (audioData.isNullOrEmpty()) continue // transcript/心跳等无音频块,跳过
|
||||||
|
val pcm = decodeAudio(audioData) ?: continue
|
||||||
|
emit(pcm)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (e: IOException) {
|
||||||
|
// 被 [cancel] 中止 → 转成协程取消信号,避免被误判为网络错误
|
||||||
|
if (call.isCanceled()) throw CancellationException("TTS request cancelled")
|
||||||
|
throw e
|
||||||
|
} finally {
|
||||||
|
if (activeCall === call) activeCall = null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 解析单个 SSE 行:`data:` 前缀 + JSON → choices[0].delta.audio.data(Base64) */
|
||||||
|
private fun parseSseLine(line: String): String? {
|
||||||
|
val trimmed = line.trim()
|
||||||
|
if (!trimmed.startsWith("data:")) return null
|
||||||
|
val payload = trimmed.removePrefix("data:").trim()
|
||||||
|
if (payload == DONE) return DONE
|
||||||
|
if (payload.isEmpty()) return null
|
||||||
|
return try {
|
||||||
|
val json = gson.fromJson(payload, JsonObject::class.java)
|
||||||
|
val delta = json.get("choices")
|
||||||
|
?.takeIf { it.isJsonArray }?.asJsonArray
|
||||||
|
?.firstOrNull()
|
||||||
|
?.takeIf { it.isJsonObject }?.asJsonObject
|
||||||
|
?.get("delta")
|
||||||
|
?.takeIf { it.isJsonObject }?.asJsonObject ?: return null
|
||||||
|
delta.get("audio")
|
||||||
|
?.takeIf { it.isJsonObject }?.asJsonObject
|
||||||
|
?.get("data")
|
||||||
|
?.takeIf { it.isJsonPrimitive }?.asString
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.w(TAG, "忽略无法解析的 SSE 块")
|
||||||
|
null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Base64 解码 audio.data 为 PCM 字节 */
|
||||||
|
private fun decodeAudio(data: String): ByteArray? = try {
|
||||||
|
Base64.getDecoder().decode(data)
|
||||||
|
} catch (e: IllegalArgumentException) {
|
||||||
|
Log.w(TAG, "audio.data 不是合法 base64,跳过")
|
||||||
|
null
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun buildRequestBody(text: String): JsonObject = JsonObject().apply {
|
||||||
|
addProperty("model", model)
|
||||||
|
val messages = JsonArray()
|
||||||
|
val userMsg = JsonObject().apply {
|
||||||
|
addProperty("role", "user")
|
||||||
|
addProperty("content", "用可爱俏皮的猫娘语气")
|
||||||
|
}
|
||||||
|
val assistantMsg = JsonObject().apply {
|
||||||
|
addProperty("role", "assistant")
|
||||||
|
addProperty("content", text)
|
||||||
|
}
|
||||||
|
messages.add(userMsg)
|
||||||
|
messages.add(assistantMsg)
|
||||||
|
add("messages", messages)
|
||||||
|
val audio = JsonObject().apply {
|
||||||
|
addProperty("format", "pcm16")
|
||||||
|
addProperty("voice", voice)
|
||||||
|
}
|
||||||
|
add("audio", audio)
|
||||||
|
addProperty("stream", true)
|
||||||
|
}
|
||||||
|
|
||||||
|
companion object {
|
||||||
|
private val JSON_MEDIA_TYPE = "application/json; charset=utf-8".toMediaType()
|
||||||
|
private const val DONE = "[DONE]"
|
||||||
|
|
||||||
|
/** 全实例共享的连接池/客户端(风格对齐 MiMoAsrClient.sharedClient,L1) */
|
||||||
|
private val sharedClient = OkHttpClient.Builder()
|
||||||
|
.connectTimeout(10, TimeUnit.SECONDS)
|
||||||
|
.readTimeout(0, TimeUnit.MILLISECONDS) // SSE:长句合成可能 60s+ 无数据
|
||||||
|
.writeTimeout(30, TimeUnit.SECONDS)
|
||||||
|
.build()
|
||||||
|
|
||||||
|
/** 规范化 baseUrl:去尾部斜杠,不以 /v1 结尾时补 /v1 */
|
||||||
|
private fun normalizeBaseUrl(raw: String): String {
|
||||||
|
val trimmed = raw.trimEnd('/')
|
||||||
|
return if (trimmed.endsWith("/v1")) trimmed else "$trimmed/v1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,181 @@
|
|||||||
|
package com.rainnya.chat.data.voice
|
||||||
|
|
||||||
|
import android.util.Log
|
||||||
|
import com.google.gson.Gson
|
||||||
|
import com.google.gson.JsonObject
|
||||||
|
import com.rainnya.chat.data.settings.AppSettings
|
||||||
|
import kotlinx.coroutines.Dispatchers
|
||||||
|
import kotlinx.coroutines.withContext
|
||||||
|
import okhttp3.OkHttpClient
|
||||||
|
import okhttp3.Request
|
||||||
|
import java.io.IOException
|
||||||
|
import java.util.concurrent.TimeUnit
|
||||||
|
|
||||||
|
private const val TAG = "RainnyaProvider"
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 服务器语音(STT/TTS)配置拉取状态,供 VoiceCallViewModel / UI 绑定。
|
||||||
|
*/
|
||||||
|
enum class VoiceConfigState {
|
||||||
|
/** 正在从 AstrBot 服务器拉取 */
|
||||||
|
Fetching,
|
||||||
|
|
||||||
|
/** 已从服务器拉取并写入(或用户已手动配置过,手动优先) */
|
||||||
|
Ready,
|
||||||
|
|
||||||
|
/** 服务器没有配置 STT 或 TTS 类型的 provider */
|
||||||
|
NotConfigured,
|
||||||
|
|
||||||
|
/** 拉取失败(网络 / 鉴权无 provider scope / 版本不支持 / 响应异常),原因见 [ProviderFetcher.lastFailureReason] */
|
||||||
|
FetchFailed,
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 从 AstrBot 服务器自动拉取语音(STT/TTS)配置(M5「复用服务端已配好的 MiMo key」)。
|
||||||
|
*
|
||||||
|
* 调用 AstrBot 的 `GET {httpBaseUrl}/api/v1/providers`,用 App 现有那把 AstrBot api_key
|
||||||
|
* ([AppSettings.apiKey],即连 WS 的那把)作 `Authorization: Bearer <key>` 鉴权;
|
||||||
|
* 响应 `data.providers[]` 原样返回 provider 配置(含明文 api_key),按 `provider_type`
|
||||||
|
* 过滤 `speech_to_text` / `text_to_speech`,命中后**仅当 [AppSettings.mimoApiKey] 当前为空**
|
||||||
|
* 才写入语音配置(手动配置优先,绝不覆盖用户手填值):
|
||||||
|
* - mimoApiKey ← provider.api_key
|
||||||
|
* - mimoBaseUrl ← provider.api_base(空则保留 AppSettings 默认 https://api.xiaomimimo.com/v1)
|
||||||
|
* - asrModel ← STT provider.model
|
||||||
|
* - ttsModel / ttsVoice ← TTS provider.model / provider.mimo-tts-voice
|
||||||
|
*
|
||||||
|
* 契约已按 AstrBot v4.27.2 源码核对(config_service.list_providers 响应结构 / 字段名 /
|
||||||
|
* CAPABILITY_TO_PROVIDER_TYPE 映射、mimo_stt/tts_api_source 配置字段)。
|
||||||
|
*
|
||||||
|
* ⚠️ 绝不把 api_key 打进日志,只落盘所需字段。
|
||||||
|
*/
|
||||||
|
class ProviderFetcher(
|
||||||
|
private val settings: AppSettings,
|
||||||
|
private val client: OkHttpClient = sharedClient,
|
||||||
|
) {
|
||||||
|
private val gson = Gson()
|
||||||
|
|
||||||
|
/** 最近一次 [VoiceConfigState.FetchFailed] 的原因描述(供 UI 展示);其他状态为 null */
|
||||||
|
@Volatile
|
||||||
|
var lastFailureReason: String? = null
|
||||||
|
private set
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 拉取服务器语音配置并(按需)写入 AppSettings。
|
||||||
|
* @return [VoiceConfigState]:Ready / NotConfigured / FetchFailed
|
||||||
|
*/
|
||||||
|
suspend fun fetchServerVoiceConfig(): VoiceConfigState = withContext(Dispatchers.IO) {
|
||||||
|
// 手动已配置优先:不拉取、不覆盖
|
||||||
|
if (settings.mimoApiKey.isNotBlank()) return@withContext VoiceConfigState.Ready
|
||||||
|
|
||||||
|
val request = Request.Builder()
|
||||||
|
.url(settings.httpBaseUrl + "/api/v1/providers")
|
||||||
|
.header("Authorization", "Bearer ${settings.apiKey}")
|
||||||
|
.get()
|
||||||
|
.build()
|
||||||
|
|
||||||
|
var state: VoiceConfigState = VoiceConfigState.FetchFailed
|
||||||
|
try {
|
||||||
|
client.newCall(request).execute().use { response ->
|
||||||
|
val body = response.body?.string().orEmpty()
|
||||||
|
state = when {
|
||||||
|
response.code == 401 || response.code == 403 -> {
|
||||||
|
lastFailureReason =
|
||||||
|
"AstrBot 的 API Key 缺少 provider 权限(HTTP ${response.code})," +
|
||||||
|
"请在 AstrBot WebUI 重新生成带 provider scope 的 Key"
|
||||||
|
VoiceConfigState.FetchFailed
|
||||||
|
}
|
||||||
|
|
||||||
|
response.code == 404 -> {
|
||||||
|
lastFailureReason =
|
||||||
|
"服务器版本不支持该接口(HTTP 404),请升级 AstrBot 后再试"
|
||||||
|
VoiceConfigState.FetchFailed
|
||||||
|
}
|
||||||
|
|
||||||
|
!response.isSuccessful -> {
|
||||||
|
lastFailureReason = "从服务器拉取语音配置失败(HTTP ${response.code})"
|
||||||
|
VoiceConfigState.FetchFailed
|
||||||
|
}
|
||||||
|
|
||||||
|
else -> parseProviders(body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (e: IOException) {
|
||||||
|
lastFailureReason = "无法连接服务器(${e.message ?: "网络错误"})"
|
||||||
|
state = VoiceConfigState.FetchFailed
|
||||||
|
}
|
||||||
|
Log.i(TAG, "fetchServerVoiceConfig -> $state")
|
||||||
|
state
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 解析 {status, data:{providers:[]}},按 provider_type 过滤 STT/TTS 并写入配置 */
|
||||||
|
private fun parseProviders(body: String): VoiceConfigState {
|
||||||
|
val json = try {
|
||||||
|
gson.fromJson(body, JsonObject::class.java)
|
||||||
|
} catch (e: Exception) {
|
||||||
|
lastFailureReason = "服务器返回无法解析的响应"
|
||||||
|
return VoiceConfigState.FetchFailed
|
||||||
|
} ?: run {
|
||||||
|
lastFailureReason = "服务器返回空响应"
|
||||||
|
return VoiceConfigState.FetchFailed
|
||||||
|
}
|
||||||
|
|
||||||
|
if (json.get("status")?.takeIf { it.isJsonPrimitive }?.asString != "ok") {
|
||||||
|
val msg = json.get("message")?.takeIf { it.isJsonPrimitive }?.asString ?: "未知错误"
|
||||||
|
lastFailureReason = "服务器返回错误:$msg"
|
||||||
|
return VoiceConfigState.FetchFailed
|
||||||
|
}
|
||||||
|
|
||||||
|
val providers = json.get("data")?.takeIf { it.isJsonObject }?.asJsonObject
|
||||||
|
?.get("providers")?.takeIf { it.isJsonArray }?.asJsonArray
|
||||||
|
if (providers == null) {
|
||||||
|
lastFailureReason = "服务器响应缺少 providers 数据"
|
||||||
|
return VoiceConfigState.FetchFailed
|
||||||
|
}
|
||||||
|
|
||||||
|
var stt: JsonObject? = null
|
||||||
|
var tts: JsonObject? = null
|
||||||
|
for (el in providers) {
|
||||||
|
val obj = el.takeIf { it.isJsonObject }?.asJsonObject ?: continue
|
||||||
|
when (obj.get("provider_type")?.takeIf { it.isJsonPrimitive }?.asString) {
|
||||||
|
"speech_to_text" -> if (stt == null) stt = obj
|
||||||
|
"text_to_speech" -> if (tts == null) tts = obj
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (stt == null && tts == null) {
|
||||||
|
lastFailureReason = null
|
||||||
|
return VoiceConfigState.NotConfigured
|
||||||
|
}
|
||||||
|
|
||||||
|
applyVoiceConfig(stt, tts)
|
||||||
|
lastFailureReason = null
|
||||||
|
return VoiceConfigState.Ready
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 把命中的 STT/TTS provider 配置写入 AppSettings(仅写非空字段,mimoApiKey 已确保为空才走到这) */
|
||||||
|
private fun applyVoiceConfig(stt: JsonObject?, tts: JsonObject?) {
|
||||||
|
stt?.let { s ->
|
||||||
|
str(s, "api_key")?.takeIf { it.isNotBlank() }?.let { settings.mimoApiKey = it }
|
||||||
|
str(s, "api_base")?.takeIf { it.isNotBlank() }?.let { settings.mimoBaseUrl = it }
|
||||||
|
str(s, "model")?.takeIf { it.isNotBlank() }?.let { settings.asrModel = it }
|
||||||
|
}
|
||||||
|
tts?.let { t ->
|
||||||
|
str(t, "api_key")?.takeIf { it.isNotBlank() }?.let { settings.mimoApiKey = it }
|
||||||
|
str(t, "api_base")?.takeIf { it.isNotBlank() }?.let { settings.mimoBaseUrl = it }
|
||||||
|
str(t, "model")?.takeIf { it.isNotBlank() }?.let { settings.ttsModel = it }
|
||||||
|
str(t, "mimo-tts-voice")?.takeIf { it.isNotBlank() }?.let { settings.ttsVoice = it }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun str(obj: JsonObject, key: String): String? =
|
||||||
|
obj.get(key)?.takeIf { it.isJsonPrimitive }?.asString
|
||||||
|
|
||||||
|
companion object {
|
||||||
|
/** 全实例共享的连接池/客户端(风格对齐 MiMoAsrClient.sharedClient) */
|
||||||
|
private val sharedClient = OkHttpClient.Builder()
|
||||||
|
.connectTimeout(10, TimeUnit.SECONDS)
|
||||||
|
.readTimeout(15, TimeUnit.SECONDS)
|
||||||
|
.writeTimeout(15, TimeUnit.SECONDS)
|
||||||
|
.build()
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
package com.rainnya.chat.data.voice
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 流式文本按句切分(M3)。
|
||||||
|
*
|
||||||
|
* [push] 接收 WS plain 增量文本,遇到完整句(以 。!?;或换行结尾)即出队返回;
|
||||||
|
* 无标点连续超过 60 字则硬切,避免 TTS 等待过长、延迟失控。残留的半句留在缓冲里,
|
||||||
|
* 等 turn 边界(`end` 事件 / streaming=false)由 [flush] 强制冲刷成收尾句;
|
||||||
|
* 打断/切会话时调 [clear] 清空残留,防止上一轮的半句与下一轮开头拼成"缝合句"送 TTS(§3.6.1)。
|
||||||
|
*/
|
||||||
|
class SentenceSplitter {
|
||||||
|
|
||||||
|
private val buffer = StringBuilder()
|
||||||
|
|
||||||
|
/** 推入一段增量文本,返回本次出队的完整句列表(残留半句留在缓冲) */
|
||||||
|
fun push(token: String): List<String> {
|
||||||
|
if (token.isEmpty()) return emptyList()
|
||||||
|
buffer.append(token)
|
||||||
|
|
||||||
|
val sentences = mutableListOf<String>()
|
||||||
|
var start = 0
|
||||||
|
var i = 0
|
||||||
|
while (i < buffer.length) {
|
||||||
|
val c = buffer[i]
|
||||||
|
val isEnd = c == '。' || c == '!' || c == '?' || c == ';' || c == '\n'
|
||||||
|
if (isEnd || i - start + 1 >= MAX_LEN) {
|
||||||
|
val segment = buffer.substring(start, i + 1).trim()
|
||||||
|
if (segment.isNotEmpty()) sentences.add(segment)
|
||||||
|
start = i + 1
|
||||||
|
}
|
||||||
|
i++
|
||||||
|
}
|
||||||
|
if (start > 0) buffer.delete(0, start)
|
||||||
|
return sentences
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 清空残留半句(打断/切会话时调用,防缝合句并入下一轮) */
|
||||||
|
fun clear() {
|
||||||
|
buffer.setLength(0)
|
||||||
|
}
|
||||||
|
|
||||||
|
/** turn 边界强制冲刷:把残留文本切成 ≤60 字的句返回并清空缓冲(§3.6.2) */
|
||||||
|
fun flush(): List<String> {
|
||||||
|
val remaining = buffer.toString().trim()
|
||||||
|
buffer.setLength(0)
|
||||||
|
if (remaining.isEmpty()) return emptyList()
|
||||||
|
|
||||||
|
val result = mutableListOf<String>()
|
||||||
|
var start = 0
|
||||||
|
while (start < remaining.length) {
|
||||||
|
val end = minOf(start + MAX_LEN, remaining.length)
|
||||||
|
result.add(remaining.substring(start, end))
|
||||||
|
start = end
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
companion object {
|
||||||
|
/** 无标点硬切上限(字) */
|
||||||
|
private const val MAX_LEN = 60
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,641 @@
|
|||||||
|
package com.rainnya.chat.data.voice
|
||||||
|
|
||||||
|
import android.util.Log
|
||||||
|
import com.rainnya.chat.data.repository.ChatRepository
|
||||||
|
import com.rainnya.chat.data.settings.AppSettings
|
||||||
|
import com.rainnya.chat.data.websocket.WsEvent
|
||||||
|
import kotlinx.coroutines.CancellationException
|
||||||
|
import kotlinx.coroutines.CoroutineScope
|
||||||
|
import kotlinx.coroutines.Job
|
||||||
|
import kotlinx.coroutines.channels.Channel
|
||||||
|
import kotlinx.coroutines.delay
|
||||||
|
import kotlinx.coroutines.flow.MutableStateFlow
|
||||||
|
import kotlinx.coroutines.flow.StateFlow
|
||||||
|
import kotlinx.coroutines.joinAll
|
||||||
|
import kotlinx.coroutines.launch
|
||||||
|
import kotlinx.coroutines.sync.Semaphore
|
||||||
|
import kotlinx.coroutines.sync.withPermit
|
||||||
|
import java.io.IOException
|
||||||
|
import java.util.concurrent.atomic.AtomicBoolean
|
||||||
|
import java.util.concurrent.atomic.AtomicInteger
|
||||||
|
import java.util.concurrent.atomic.AtomicLong
|
||||||
|
|
||||||
|
private const val TAG = "RainnyaVoice"
|
||||||
|
|
||||||
|
/** 语音通话状态机(M0/M1/M3) */
|
||||||
|
sealed interface CallState {
|
||||||
|
object Idle : CallState
|
||||||
|
object Recording : CallState
|
||||||
|
object Asr : CallState
|
||||||
|
object WaitingReply : CallState
|
||||||
|
object Playing : CallState
|
||||||
|
object Interrupted : CallState // M4+ 预留
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 回复管线内部指令:合成一句 / turn 边界冲刷结束 */
|
||||||
|
private sealed interface ReplyCmd {
|
||||||
|
data class Synthesize(val text: String) : ReplyCmd
|
||||||
|
object TurnEnd : ReplyCmd
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 语音通话引擎(观察者,不是 ChatRepository 的改版)。
|
||||||
|
*
|
||||||
|
* 流程:
|
||||||
|
* - M0/M1:Recording → VAD 结束 → Asr → 识别成功 → repository.sendVoiceMessage(文本) → WaitingReply
|
||||||
|
* - M3:sendVoiceMessage 成功 → WaitingReply(等待雨喵回复…)
|
||||||
|
* → 订阅 repository.wsEvents 的 plain 文本流(按 message_id 去重、只处理当前 turn)
|
||||||
|
* → SentenceSplitter 出句 → 逐句 MiMoTtsClient 流式合成(有界并行 2 + PCM 带 seq 按序重排)
|
||||||
|
* → AudioPlaybackEngine.enqueue + play → Playing(播放中…)
|
||||||
|
* → `end`/streaming=false 事件 = turn 边界 → splitter.flush() 收尾 → 播完 → 回 Idle
|
||||||
|
*
|
||||||
|
* 管线边界语义(§3.6):
|
||||||
|
* 1. 打断 = 原子操作:playback.stop() + splitter.clear() + 取消在途 TTS SSE + 标记本轮不再合成;
|
||||||
|
* 2. turn 边界冲刷:依赖 `end` 事件,streaming=false 的 plain 也作边界兜底;
|
||||||
|
* 3. 句序防乱序:句间重叠管线 + PCM 块带 seq 由 AudioPlaybackEngine 按序重排(有界并行 2);
|
||||||
|
* 4. 驱动源:只订阅 wsEvents 增量(plain/end),不订阅 repository.messages 全量列表。
|
||||||
|
*
|
||||||
|
* 错误兜底(§6):TTS 失败/断流 → 停止播放 + 文字气泡兜底(聊天页已有),回 Idle 不崩溃;
|
||||||
|
* WAITING_REPLY 20s 无首包 → 提示网络异常。本轮不做 barge-in 麦克风打断(M4)。
|
||||||
|
*/
|
||||||
|
class VoiceCallEngine(
|
||||||
|
private val scope: CoroutineScope,
|
||||||
|
private val settings: AppSettings,
|
||||||
|
private val repository: ChatRepository,
|
||||||
|
) {
|
||||||
|
private val _state = MutableStateFlow<CallState>(CallState.Idle)
|
||||||
|
val state: StateFlow<CallState> = _state
|
||||||
|
|
||||||
|
private val _statusText = MutableStateFlow("空闲")
|
||||||
|
val statusText: StateFlow<String> = _statusText
|
||||||
|
|
||||||
|
private val _transcribedText = MutableStateFlow<String?>(null)
|
||||||
|
val transcribedText: StateFlow<String?> = _transcribedText
|
||||||
|
|
||||||
|
private val _lastError = MutableStateFlow<String?>(null)
|
||||||
|
val lastError: StateFlow<String?> = _lastError
|
||||||
|
|
||||||
|
/** 是否正在录音/识别(供 UI 生命周期钩子判断是否需要 stopTalking) */
|
||||||
|
private val _isRecording = MutableStateFlow(false)
|
||||||
|
val isRecording: StateFlow<Boolean> = _isRecording
|
||||||
|
|
||||||
|
private val recorder = AudioRecorder()
|
||||||
|
|
||||||
|
private var micPermissionGranted = false
|
||||||
|
private var talkJob: Job? = null
|
||||||
|
|
||||||
|
/** 当前 in-flight 的 ASR 客户端,供 stopTalking/hangUp 中止阻塞中的 HTTP(M2) */
|
||||||
|
private var currentAsr: MiMoAsrClient? = null
|
||||||
|
|
||||||
|
// ===== M3 TTS 播放管线 =====
|
||||||
|
|
||||||
|
private val playback = AudioPlaybackEngine()
|
||||||
|
private val splitter = SentenceSplitter()
|
||||||
|
|
||||||
|
/** 句子命令通道:watcher 产句,pipeline 消费;TurnEnd 作 turn 边界哨兵 */
|
||||||
|
private val sentenceChannel = Channel<ReplyCmd>(Channel.UNLIMITED)
|
||||||
|
|
||||||
|
/** TTS 有界并行度上限(v1=2,防句间乱序过远,§3.6.3) */
|
||||||
|
private val ttsSemaphore = Semaphore(MAX_TTS_PARALLELISM)
|
||||||
|
|
||||||
|
private val seqCounter = AtomicInteger(0)
|
||||||
|
private val totalQueuedBytes = AtomicLong(0)
|
||||||
|
private val activeTts = mutableSetOf<MiMoTtsClient>()
|
||||||
|
private val currentTurnActive = AtomicBoolean(false)
|
||||||
|
private val turnEnded = AtomicBoolean(false)
|
||||||
|
private val synthesisFailed = AtomicBoolean(false)
|
||||||
|
@Volatile
|
||||||
|
private var playStarted = false
|
||||||
|
|
||||||
|
private var watcherJob: Job? = null
|
||||||
|
private var pipelineJob: Job? = null
|
||||||
|
private var waitingTimeoutJob: Job? = null
|
||||||
|
private var turnWatchdogJob: Job? = null
|
||||||
|
|
||||||
|
/** M4:播放态 barge-in 麦克风监听协程(进入 Playing 启动、离开 Playing 取消) */
|
||||||
|
private var bargeInJob: Job? = null
|
||||||
|
|
||||||
|
init {
|
||||||
|
// 随引擎 scope 取消(如 ViewModel 清理)释放播放资源,避免 AudioTrack/线程泄漏
|
||||||
|
scope.launch {
|
||||||
|
try {
|
||||||
|
while (true) delay(Long.MAX_VALUE)
|
||||||
|
} catch (e: CancellationException) {
|
||||||
|
playback.release()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 麦克风权限已授予(由 ViewModel 在运行时权限结果回调时调用) */
|
||||||
|
fun onMicPermissionGranted() {
|
||||||
|
micPermissionGranted = true
|
||||||
|
Log.d(TAG, "Mic permission granted")
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 开始说话:进入 RECORDING;VAD 静音 600ms 自动结束;45s 硬切 */
|
||||||
|
fun startTalking() {
|
||||||
|
when (_state.value) {
|
||||||
|
CallState.Idle -> { /* 正常开始一轮 */ }
|
||||||
|
CallState.Playing, CallState.WaitingReply -> {
|
||||||
|
// M4 手动打断:点麦克风即打断播放/等待,先执行打断原子操作,再进入新一轮录音
|
||||||
|
Log.d(TAG, "startTalking(手动打断回复)")
|
||||||
|
interruptToRecording("手动打断")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
else -> {
|
||||||
|
Log.w(TAG, "startTalking ignored: state=${_state.value}")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!settings.isVoiceConfigured) {
|
||||||
|
fail("请先在设置中配置 MiMo 语音(API Key)")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if (!micPermissionGranted) {
|
||||||
|
fail("需要麦克风权限")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
Log.d(TAG, "startTalking")
|
||||||
|
_lastError.value = null
|
||||||
|
startTalkJob()
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 手动结束录音(长按抬手等场景);Recording/Asr 取消录音与 ASR;WaitingReply/Playing 执行打断原子操作 */
|
||||||
|
fun stopTalking() {
|
||||||
|
when (_state.value) {
|
||||||
|
CallState.Idle -> return
|
||||||
|
CallState.Recording, CallState.Asr -> {
|
||||||
|
Log.d(TAG, "stopTalking(录制/识别中)")
|
||||||
|
talkJob?.cancel()
|
||||||
|
talkJob = null
|
||||||
|
currentAsr?.cancel()
|
||||||
|
currentAsr = null
|
||||||
|
_state.value = CallState.Idle
|
||||||
|
_statusText.value = "空闲"
|
||||||
|
_isRecording.value = false
|
||||||
|
}
|
||||||
|
CallState.WaitingReply, CallState.Playing -> {
|
||||||
|
Log.d(TAG, "stopTalking(打断回复)")
|
||||||
|
abortTurn(null)
|
||||||
|
}
|
||||||
|
CallState.Interrupted -> {
|
||||||
|
// M4 打断短暂态:取消等待中的新一轮录音,回 Idle
|
||||||
|
stopBargeIn()
|
||||||
|
talkJob?.cancel()
|
||||||
|
talkJob = null
|
||||||
|
_state.value = CallState.Idle
|
||||||
|
_statusText.value = "空闲"
|
||||||
|
_isRecording.value = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 挂断:回 Idle,清空本轮状态,中止录音/ASR/在途 TTS/barge-in 监听 */
|
||||||
|
fun hangUp() {
|
||||||
|
Log.d(TAG, "hangUp")
|
||||||
|
stopBargeIn()
|
||||||
|
when (_state.value) {
|
||||||
|
CallState.Recording, CallState.Asr -> {
|
||||||
|
talkJob?.cancel()
|
||||||
|
talkJob = null
|
||||||
|
currentAsr?.cancel()
|
||||||
|
currentAsr = null
|
||||||
|
}
|
||||||
|
CallState.WaitingReply, CallState.Playing -> abortTurn(null)
|
||||||
|
CallState.Interrupted -> {
|
||||||
|
// M4 打断短暂态:取消等待中的新一轮录音
|
||||||
|
talkJob?.cancel()
|
||||||
|
talkJob = null
|
||||||
|
}
|
||||||
|
else -> {}
|
||||||
|
}
|
||||||
|
_state.value = CallState.Idle
|
||||||
|
_statusText.value = "空闲"
|
||||||
|
_transcribedText.value = null
|
||||||
|
_lastError.value = null
|
||||||
|
_isRecording.value = false
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun startTalkJob() {
|
||||||
|
talkJob?.cancel()
|
||||||
|
talkJob = scope.launch {
|
||||||
|
runRecordingLoop()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 录音→VAD→ASR→发送→启动回复管线 的完整一轮(M0/M1/M2/M3 链路;M4 打断后复用进入新一轮录音) */
|
||||||
|
private suspend fun runRecordingLoop() {
|
||||||
|
var emptyRetries = 1 // 空文本自动重录次数上限(1 次)
|
||||||
|
while (true) {
|
||||||
|
_state.value = CallState.Recording
|
||||||
|
_isRecording.value = true
|
||||||
|
_statusText.value = "录音中…"
|
||||||
|
|
||||||
|
val wav: ByteArray? = try {
|
||||||
|
recorder.recordWithVad()
|
||||||
|
} catch (e: CancellationException) {
|
||||||
|
throw e // 打断/挂断的取消信号,不当作错误
|
||||||
|
} catch (e: SecurityException) {
|
||||||
|
fail("权限被拒,无法录音")
|
||||||
|
return
|
||||||
|
} catch (e: IllegalStateException) {
|
||||||
|
fail("录音设备被占用,请稍后再试")
|
||||||
|
return
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.e(TAG, "Record failed", e)
|
||||||
|
fail("录音失败,请重试")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if (wav == null) {
|
||||||
|
// 45s 内全程未检测到语音 → 丢弃、不发 ASR
|
||||||
|
fail("没听清,请再说一次")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
_state.value = CallState.Asr
|
||||||
|
_statusText.value = "识别中…"
|
||||||
|
|
||||||
|
val asr = asrClient()
|
||||||
|
currentAsr = asr
|
||||||
|
val text: String = try {
|
||||||
|
asr.transcribe(wav)
|
||||||
|
} catch (e: CancellationException) {
|
||||||
|
throw e
|
||||||
|
} catch (e: IOException) {
|
||||||
|
Log.e(TAG, "ASR network error", e)
|
||||||
|
fail("网络异常,请重试")
|
||||||
|
return
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.e(TAG, "ASR failed", e)
|
||||||
|
fail("识别失败,请重试")
|
||||||
|
return
|
||||||
|
} finally {
|
||||||
|
currentAsr = null
|
||||||
|
}
|
||||||
|
|
||||||
|
val trimmed = text.trim()
|
||||||
|
if (trimmed.isEmpty()) {
|
||||||
|
if (emptyRetries > 0) {
|
||||||
|
emptyRetries--
|
||||||
|
_lastError.value = "没听清,再试一次"
|
||||||
|
continue // 自动重录一次
|
||||||
|
}
|
||||||
|
fail("没听清,请再说一次")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
_transcribedText.value = trimmed
|
||||||
|
// 先启动回复管线(订阅 wsEvents)再发送,保证不丢首包
|
||||||
|
startReplyPipeline()
|
||||||
|
val sent = repository.sendVoiceMessage(trimmed)
|
||||||
|
if (!sent) {
|
||||||
|
Log.w(TAG, "sendVoiceMessage failed: 未连接")
|
||||||
|
abortTurn(null)
|
||||||
|
fail("未连接,消息未发送")
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ===== M3 回复管线 =====
|
||||||
|
|
||||||
|
/** 启动一轮回复:重置状态 → WaitingReply → 订阅 wsEvents → 等首包/超时 */
|
||||||
|
private fun startReplyPipeline() {
|
||||||
|
Log.d(TAG, "startReplyPipeline")
|
||||||
|
resetTurnState()
|
||||||
|
currentTurnActive.set(true)
|
||||||
|
turnEnded.set(false)
|
||||||
|
synthesisFailed.set(false)
|
||||||
|
splitter.clear()
|
||||||
|
// 清掉可能残留的上轮指令,防止串句
|
||||||
|
while (sentenceChannel.tryReceive().isSuccess) {
|
||||||
|
// drain
|
||||||
|
}
|
||||||
|
_state.value = CallState.WaitingReply
|
||||||
|
_statusText.value = "等待雨喵回复…"
|
||||||
|
_isRecording.value = false
|
||||||
|
startWatcher()
|
||||||
|
startPipeline()
|
||||||
|
startWaitingTimeout()
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 订阅 wsEvents 的 plain/end 增量流(§3.6.4),按 message_id 去重,
|
||||||
|
* 只处理当前 turn 的文本;`end`/streaming=false 作为 turn 边界触发冲刷。
|
||||||
|
*/
|
||||||
|
private fun startWatcher() {
|
||||||
|
watcherJob?.cancel()
|
||||||
|
watcherJob = scope.launch {
|
||||||
|
var turnMessageId: String? = null
|
||||||
|
repository.wsEvents.collect { event ->
|
||||||
|
if (!currentTurnActive.get()) return@collect
|
||||||
|
if (event !is WsEvent.MessageReceived) return@collect
|
||||||
|
val msg = event.msg
|
||||||
|
when (msg.type) {
|
||||||
|
"plain" -> {
|
||||||
|
val text = msg.data?.toString() ?: return@collect
|
||||||
|
val trimmed = text.trim()
|
||||||
|
if (trimmed.isEmpty()) return@collect
|
||||||
|
// 工具调用/结果/推理文本段不进入 TTS(不发送语音);
|
||||||
|
// chatcmpl-tool- 启发式仅作 chain_type 缺失时的兜底
|
||||||
|
when (msg.chain_type) {
|
||||||
|
"tool_call", "tool_call_result", "reasoning" -> return@collect
|
||||||
|
else -> {
|
||||||
|
if (trimmed.startsWith("{") && trimmed.contains("chatcmpl-tool-")) {
|
||||||
|
return@collect
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (turnMessageId == null) {
|
||||||
|
// 首包:进入播放态,取消 20s 超时,启动 turn 看门狗(防 end 丢失导致卡死)
|
||||||
|
turnMessageId = msg.message_id
|
||||||
|
waitingTimeoutJob?.cancel()
|
||||||
|
_state.value = CallState.Playing
|
||||||
|
_statusText.value = "播放中…"
|
||||||
|
startTurnWatchdog()
|
||||||
|
// M4:播放态持续监听麦克风,用户一开口(barge-in)就打断播放进入新一轮录音
|
||||||
|
startBargeIn(BARGE_IN_PLAYING_THRESHOLD)
|
||||||
|
} else if (msg.message_id != null && msg.message_id != turnMessageId) {
|
||||||
|
return@collect // 其他流的增量,忽略
|
||||||
|
}
|
||||||
|
val isStreaming = msg.streaming ?: true
|
||||||
|
splitter.push(text).forEach { sentenceChannel.send(ReplyCmd.Synthesize(it)) }
|
||||||
|
if (!isStreaming) {
|
||||||
|
sendTurnEnd() // streaming=false 的 plain 也作为 turn 边界(§3.6.2)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"end" -> {
|
||||||
|
// turn 边界(§3.6.2):依赖 AstrBot end 事件
|
||||||
|
if (turnMessageId == null || msg.message_id == null || msg.message_id == turnMessageId) {
|
||||||
|
sendTurnEnd()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** turn 边界冲刷:把残留收尾句强制送出 + 投递 TurnEnd 哨兵(防重复) */
|
||||||
|
private suspend fun sendTurnEnd() {
|
||||||
|
if (!turnEnded.getAndSet(true)) {
|
||||||
|
splitter.flush().forEach { sentenceChannel.send(ReplyCmd.Synthesize(it)) }
|
||||||
|
sentenceChannel.send(ReplyCmd.TurnEnd)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 消费句子命令:有界并行合成 → 等全部合成完成 → 等播放排空 → 回 Idle */
|
||||||
|
private fun startPipeline() {
|
||||||
|
pipelineJob?.cancel()
|
||||||
|
pipelineJob = scope.launch {
|
||||||
|
val children = mutableListOf<Job>()
|
||||||
|
for (cmd in sentenceChannel) {
|
||||||
|
if (!currentTurnActive.get()) return@launch
|
||||||
|
when (cmd) {
|
||||||
|
is ReplyCmd.Synthesize -> {
|
||||||
|
if (cmd.text.isNotBlank() && !synthesisFailed.get()) {
|
||||||
|
children += launch { synthesizeAndEnqueue(cmd.text) }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
is ReplyCmd.TurnEnd -> break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!currentTurnActive.get()) return@launch
|
||||||
|
children.joinAll() // 等所有合成(含收尾句)完成,totalQueuedBytes 定稿
|
||||||
|
if (!currentTurnActive.get()) return@launch
|
||||||
|
waitForPlaybackDrain()
|
||||||
|
if (!currentTurnActive.get()) return@launch
|
||||||
|
playback.stop()
|
||||||
|
resetTurnState()
|
||||||
|
currentTurnActive.set(false)
|
||||||
|
backToIdle()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 单句流式合成:SSE → PCM 块带 seq 入队(句间重叠,AudioPlaybackEngine 按序重排) */
|
||||||
|
private suspend fun synthesizeAndEnqueue(rawText: String) {
|
||||||
|
val text = sanitizeForTts(rawText)
|
||||||
|
if (text.isEmpty()) return
|
||||||
|
ttsSemaphore.withPermit {
|
||||||
|
val tts = ttsClient()
|
||||||
|
synchronized(activeTts) { activeTts.add(tts) }
|
||||||
|
try {
|
||||||
|
tts.synthesizeStream(text).collect { chunk ->
|
||||||
|
// 打断/失败标记 → 本轮不再合成(§3.6.1 ④)
|
||||||
|
if (!currentTurnActive.get() || synthesisFailed.get()) {
|
||||||
|
throw CancellationException("turn interrupted")
|
||||||
|
}
|
||||||
|
val seq = seqCounter.incrementAndGet()
|
||||||
|
totalQueuedBytes.addAndGet(chunk.size.toLong())
|
||||||
|
playback.enqueue(seq, chunk)
|
||||||
|
if (!playStarted) {
|
||||||
|
playStarted = true
|
||||||
|
playback.play()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (e: CancellationException) {
|
||||||
|
throw e
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.e(TAG, "TTS 合成失败:$text", e)
|
||||||
|
// §6:TTS 失败 → 停止播放,文字气泡仍在(聊天页已有),不崩溃
|
||||||
|
if (currentTurnActive.get() && !synthesisFailed.get()) {
|
||||||
|
synthesisFailed.set(true)
|
||||||
|
abortTurn("语音播放失败,文字已显示")
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
synchronized(activeTts) { activeTts.remove(tts) }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 等播放排空:全部入队字节播完即结束(带兜底超时) */
|
||||||
|
private suspend fun waitForPlaybackDrain() {
|
||||||
|
val deadline = System.currentTimeMillis() + PLAYBACK_DRAIN_TIMEOUT_MS
|
||||||
|
while (System.currentTimeMillis() < deadline) {
|
||||||
|
if (!currentTurnActive.get()) return
|
||||||
|
val queued = totalQueuedBytes.get()
|
||||||
|
if (queued == 0L) return // 本轮无音频(空回复等)
|
||||||
|
if (playback.playedBytesCount() >= queued) return
|
||||||
|
delay(30)
|
||||||
|
}
|
||||||
|
Log.w(TAG, "播放排空超时,强制结束本轮")
|
||||||
|
}
|
||||||
|
|
||||||
|
/** WAITING_REPLY 20s 无首包 → 提示网络异常(§6/P1) */
|
||||||
|
private fun startWaitingTimeout() {
|
||||||
|
waitingTimeoutJob?.cancel()
|
||||||
|
waitingTimeoutJob = scope.launch {
|
||||||
|
delay(WAITING_REPLY_TIMEOUT_MS)
|
||||||
|
if (currentTurnActive.get() && _state.value == CallState.WaitingReply) {
|
||||||
|
Log.w(TAG, "WAITING_REPLY 20s 无首包")
|
||||||
|
abortTurn("雨喵回复超时,请检查网络")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* turn 看门狗:首包后若 end/streaming=false 迟迟不到(断流),45s 后强制冲刷收尾,
|
||||||
|
* 避免状态卡在 Playing 永远不结束。
|
||||||
|
*/
|
||||||
|
private fun startTurnWatchdog() {
|
||||||
|
turnWatchdogJob?.cancel()
|
||||||
|
turnWatchdogJob = scope.launch {
|
||||||
|
delay(TURN_WATCHDOG_MS)
|
||||||
|
if (currentTurnActive.get() && !turnEnded.get()) {
|
||||||
|
Log.w(TAG, "turn 看门狗:end 未到达,强制冲刷收尾")
|
||||||
|
sendTurnEnd()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 打断原子操作(§3.6.1,四件事):
|
||||||
|
* ① playback.stop()(清 PCM 队列 + pause + flush)
|
||||||
|
* ② splitter.clear()(清残留半句,防"缝合句")
|
||||||
|
* ③ 取消在途 TTS SSE(activeTts 逐个 cancel)
|
||||||
|
* ④ currentTurnActive=false 标记本轮不再合成(在途合成协程随即退出)
|
||||||
|
* 仅当本轮仍 active 时执行(幂等);中止后回 Idle。
|
||||||
|
*/
|
||||||
|
private fun abortTurn(message: String?) {
|
||||||
|
if (!currentTurnActive.getAndSet(false)) return
|
||||||
|
Log.w(TAG, "abortTurn: $message")
|
||||||
|
doInterruptAtomic()
|
||||||
|
_lastError.value = message
|
||||||
|
_state.value = CallState.Idle
|
||||||
|
_statusText.value = "空闲"
|
||||||
|
_isRecording.value = false
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 打断原子操作执行体(§3.6.1):无条件执行,供 abortTurn 与 M4 打断共用(见 interruptToRecording) */
|
||||||
|
private fun doInterruptAtomic() {
|
||||||
|
playback.stop()
|
||||||
|
splitter.clear()
|
||||||
|
synchronized(activeTts) { activeTts.toList() }.forEach { it.cancel() }
|
||||||
|
currentTurnActive.set(false)
|
||||||
|
watcherJob?.cancel()
|
||||||
|
waitingTimeoutJob?.cancel()
|
||||||
|
turnWatchdogJob?.cancel()
|
||||||
|
pipelineJob?.cancel()
|
||||||
|
resetTurnState()
|
||||||
|
}
|
||||||
|
|
||||||
|
// ===== M4 Barge-in 打断 =====
|
||||||
|
|
||||||
|
/** 播放态启动 barge-in 监听:播放 → 用户开口(RMS 持续 ≥300ms 超阈值)→ 打断 → 新一轮录音 */
|
||||||
|
private fun startBargeIn(threshold: Float) {
|
||||||
|
stopBargeIn()
|
||||||
|
bargeInJob = scope.launch {
|
||||||
|
try {
|
||||||
|
recorder.listenForBargeIn(threshold)
|
||||||
|
onBargeInTriggered()
|
||||||
|
} catch (e: CancellationException) {
|
||||||
|
throw e
|
||||||
|
} catch (e: Exception) {
|
||||||
|
// 麦克风被占用/权限异常:静默降级,不影响播放(barge-in 是增强功能,非核心链路)
|
||||||
|
Log.w(TAG, "barge-in 监听失败", e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 停止 barge-in 监听(离开 Playing / 打断 / 挂断 / 自然播完时调用) */
|
||||||
|
private fun stopBargeIn() {
|
||||||
|
bargeInJob?.cancel()
|
||||||
|
bargeInJob = null
|
||||||
|
}
|
||||||
|
|
||||||
|
/** barge-in 触发(用户开口打断播放)→ 打断原子操作 → Interrupted → 新一轮录音 */
|
||||||
|
private fun onBargeInTriggered() {
|
||||||
|
val s = _state.value
|
||||||
|
if (s != CallState.Playing && s != CallState.WaitingReply) {
|
||||||
|
Log.w(TAG, "barge-in 触发但已离开播放态(state=$s),忽略")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
Log.d(TAG, "barge-in 触发:用户开口打断播放")
|
||||||
|
interruptToRecording("barge-in 打断")
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* M4 打断入口:① 打断原子操作(停播放 + 清分句 + 取消在途 TTS + 标记不再合成)
|
||||||
|
* ② Interrupted 短暂中间态(statusText「已打断,随时可以重新开始」)
|
||||||
|
* ③ 直接进入新一轮录音(复用 runRecordingLoop)
|
||||||
|
*
|
||||||
|
* 与 M3 状态机的协作:打断后跳过"等播放排空回 Idle",直接进新录音;
|
||||||
|
* 本轮 LLM 流式文本仍进气泡(ChatRepository 已处理),引擎不再对其合成语音(currentTurnActive=false)。
|
||||||
|
*/
|
||||||
|
private fun interruptToRecording(reason: String) {
|
||||||
|
Log.w(TAG, "interruptToRecording: $reason")
|
||||||
|
doInterruptAtomic() // 无条件执行:即使正在播放排空尾巴,也停掉管线,防旧 pipeline 的 backToIdle 覆盖新状态
|
||||||
|
_lastError.value = null
|
||||||
|
_state.value = CallState.Interrupted
|
||||||
|
_statusText.value = "已打断,随时可以重新开始"
|
||||||
|
_isRecording.value = false
|
||||||
|
talkJob?.cancel()
|
||||||
|
talkJob = scope.launch {
|
||||||
|
delay(INTERRUPT_TO_RECORD_DELAY_MS) // 短暂展示 Interrupted 态,随即进入新一轮录音
|
||||||
|
runRecordingLoop()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 每轮开始前重置播放/计数状态(seq 从 0 起、清残留播放队列),并停止 barge-in 监听 */
|
||||||
|
private fun resetTurnState() {
|
||||||
|
seqCounter.set(0)
|
||||||
|
totalQueuedBytes.set(0)
|
||||||
|
playStarted = false
|
||||||
|
playback.stop()
|
||||||
|
stopBargeIn()
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 每次调用用最新设置构造 TTS 客户端 */
|
||||||
|
private fun ttsClient(): MiMoTtsClient =
|
||||||
|
MiMoTtsClient(
|
||||||
|
baseUrl = settings.mimoBaseUrl,
|
||||||
|
apiKey = settings.mimoApiKey,
|
||||||
|
model = settings.ttsModel,
|
||||||
|
voice = settings.ttsVoice,
|
||||||
|
)
|
||||||
|
|
||||||
|
/** 每次调用用最新设置构造 ASR 客户端(model 注入 settings.asrModel) */
|
||||||
|
private fun asrClient(): MiMoAsrClient =
|
||||||
|
MiMoAsrClient(baseUrl = settings.mimoBaseUrl, apiKey = settings.mimoApiKey).apply {
|
||||||
|
model = settings.asrModel
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 轻量清理 markdown 记号,避免雨喵读出 **、`、链接等字符(只影响 TTS 输入,不影响文字气泡) */
|
||||||
|
private fun sanitizeForTts(raw: String): String {
|
||||||
|
var s = raw
|
||||||
|
.replace(Regex("""\*\*|__|~~|`+"""), "")
|
||||||
|
.replace(Regex("""\[([^\]]*)\]\([^)]*\)"""), "$1")
|
||||||
|
.replace(Regex("""(?m)^\s*[#>]\s*"""), "")
|
||||||
|
return s.trim()
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun fail(message: String) {
|
||||||
|
Log.w(TAG, "Voice fail: $message")
|
||||||
|
_lastError.value = message
|
||||||
|
_state.value = CallState.Idle
|
||||||
|
_statusText.value = "空闲"
|
||||||
|
_isRecording.value = false
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun backToIdle() {
|
||||||
|
// M4:若已被打断/已进入新一轮录音(状态不再是 Playing),旧管线不得覆盖状态
|
||||||
|
if (_state.value != CallState.Playing) return
|
||||||
|
_state.value = CallState.Idle
|
||||||
|
_statusText.value = "空闲"
|
||||||
|
_lastError.value = null
|
||||||
|
_isRecording.value = false
|
||||||
|
}
|
||||||
|
|
||||||
|
companion object {
|
||||||
|
private const val MAX_TTS_PARALLELISM = 2
|
||||||
|
private const val WAITING_REPLY_TIMEOUT_MS = 20_000L
|
||||||
|
private const val TURN_WATCHDOG_MS = 45_000L
|
||||||
|
private const val PLAYBACK_DRAIN_TIMEOUT_MS = 15_000L
|
||||||
|
|
||||||
|
// ===== M4 Barge-in =====
|
||||||
|
/** 播放态 barge-in 阈值(RMS 0..32767):正常 VAD 固定下限(800)的 2 倍,防回声误触发(§6/P1;不做 AEC) */
|
||||||
|
private const val BARGE_IN_PLAYING_THRESHOLD = 1600f
|
||||||
|
/** 打断后短暂展示 Interrupted 态再进入新一轮录音的间隔(ms) */
|
||||||
|
private const val INTERRUPT_TO_RECORD_DELAY_MS = 300L
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -78,6 +78,7 @@ class AstrBotWsClient(
|
|||||||
code = json.get("code")?.asString,
|
code = json.get("code")?.asString,
|
||||||
attachment_id = json.get("attachment_id")?.asString,
|
attachment_id = json.get("attachment_id")?.asString,
|
||||||
url = json.get("url")?.asString,
|
url = json.get("url")?.asString,
|
||||||
|
chain_type = json.get("chain_type")?.asString,
|
||||||
)
|
)
|
||||||
scope.launch { _events.send(WsEvent.MessageReceived(msg)) }
|
scope.launch { _events.send(WsEvent.MessageReceived(msg)) }
|
||||||
} catch (e: Exception) {
|
} catch (e: Exception) {
|
||||||
|
|||||||
@@ -110,7 +110,7 @@ fun MessageBubble(
|
|||||||
.background(bubbleColor)
|
.background(bubbleColor)
|
||||||
.combinedClickable(
|
.combinedClickable(
|
||||||
onClick = { },
|
onClick = { },
|
||||||
onLongClick = { onCopy(message.content) },
|
onLongClick = { onCopy(copyPayload(message)) },
|
||||||
)
|
)
|
||||||
.padding(horizontal = 14.dp, vertical = 10.dp),
|
.padding(horizontal = 14.dp, vertical = 10.dp),
|
||||||
) {
|
) {
|
||||||
@@ -160,11 +160,31 @@ private fun BubbleContent(
|
|||||||
) {
|
) {
|
||||||
val isStreaming = message.streaming
|
val isStreaming = message.streaming
|
||||||
|
|
||||||
|
// 工具调用卡片:独立于正文渲染;content 为空时不渲染空白正文
|
||||||
|
if (message.toolCall != null) {
|
||||||
|
ToolCallCard(toolCallJson = message.toolCall)
|
||||||
|
if (message.content.isNotBlank()) {
|
||||||
|
Spacer(Modifier.height(8.dp))
|
||||||
|
MessageBody(message = message, textColor = textColor)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
if (isStreaming && message.content.isEmpty()) {
|
if (isStreaming && message.content.isEmpty()) {
|
||||||
PulseLoadingBar(textColor = textColor)
|
PulseLoadingBar(textColor = textColor)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
MessageBody(message = message, textColor = textColor)
|
||||||
|
}
|
||||||
|
|
||||||
|
@Composable
|
||||||
|
private fun MessageBody(
|
||||||
|
message: ChatMessage,
|
||||||
|
textColor: Color,
|
||||||
|
) {
|
||||||
|
val isStreaming = message.streaming
|
||||||
|
|
||||||
var expanded by remember(message.id) { mutableStateOf(false) }
|
var expanded by remember(message.id) { mutableStateOf(false) }
|
||||||
val shouldCollapse = !isStreaming && message.content.length > COLLAPSE_THRESHOLD
|
val shouldCollapse = !isStreaming && message.content.length > COLLAPSE_THRESHOLD
|
||||||
val displayText = if (shouldCollapse && !expanded) {
|
val displayText = if (shouldCollapse && !expanded) {
|
||||||
@@ -321,6 +341,12 @@ private fun SystemMessageBubble(message: ChatMessage, modifier: Modifier = Modif
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private fun copyPayload(message: ChatMessage): String = when {
|
||||||
|
message.content.isNotBlank() -> message.content
|
||||||
|
!message.toolCall.isNullOrBlank() -> message.toolCall!!
|
||||||
|
else -> ""
|
||||||
|
}
|
||||||
|
|
||||||
private fun timestampString(timestamp: Long): String {
|
private fun timestampString(timestamp: Long): String {
|
||||||
val now = System.currentTimeMillis()
|
val now = System.currentTimeMillis()
|
||||||
val sdf = if (now - timestamp < 86400000L) {
|
val sdf = if (now - timestamp < 86400000L) {
|
||||||
@@ -356,3 +382,18 @@ fun MessageBubbleAssistantPreview() {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Preview(showBackground = true)
|
||||||
|
@Composable
|
||||||
|
fun MessageBubbleToolCallPreview() {
|
||||||
|
RainnyaTheme {
|
||||||
|
MessageBubble(
|
||||||
|
message = ChatMessage(
|
||||||
|
id = "3",
|
||||||
|
content = "",
|
||||||
|
role = MessageRole.ASSISTANT,
|
||||||
|
toolCall = """{"id":"chatcmpl-tool-t1","name":"astrbot_web_search","args":"{\"query\":\"今天的新闻\"}","result":"{\"items\":[{\"title\":\"新闻一\"}]}","ts":1756300000000,"finished_ts":1756300001200}""",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,399 @@
|
|||||||
|
package com.rainnya.chat.ui.components
|
||||||
|
|
||||||
|
import androidx.compose.animation.AnimatedVisibility
|
||||||
|
import androidx.compose.animation.core.FastOutSlowInEasing
|
||||||
|
import androidx.compose.animation.core.RepeatMode
|
||||||
|
import androidx.compose.animation.core.animateFloat
|
||||||
|
import androidx.compose.animation.core.animateFloatAsState
|
||||||
|
import androidx.compose.animation.core.infiniteRepeatable
|
||||||
|
import androidx.compose.animation.core.rememberInfiniteTransition
|
||||||
|
import androidx.compose.animation.core.tween
|
||||||
|
import androidx.compose.animation.expandVertically
|
||||||
|
import androidx.compose.animation.fadeIn
|
||||||
|
import androidx.compose.animation.fadeOut
|
||||||
|
import androidx.compose.animation.shrinkVertically
|
||||||
|
import androidx.compose.foundation.background
|
||||||
|
import androidx.compose.foundation.border
|
||||||
|
import androidx.compose.foundation.clickable
|
||||||
|
import androidx.compose.foundation.layout.Box
|
||||||
|
import androidx.compose.foundation.layout.Column
|
||||||
|
import androidx.compose.foundation.layout.Row
|
||||||
|
import androidx.compose.foundation.layout.Spacer
|
||||||
|
import androidx.compose.foundation.layout.fillMaxWidth
|
||||||
|
import androidx.compose.foundation.layout.height
|
||||||
|
import androidx.compose.foundation.layout.heightIn
|
||||||
|
import androidx.compose.foundation.layout.padding
|
||||||
|
import androidx.compose.foundation.layout.size
|
||||||
|
import androidx.compose.foundation.layout.width
|
||||||
|
import androidx.compose.foundation.rememberScrollState
|
||||||
|
import androidx.compose.foundation.shape.CircleShape
|
||||||
|
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||||
|
import androidx.compose.foundation.verticalScroll
|
||||||
|
import androidx.compose.material.icons.Icons
|
||||||
|
import androidx.compose.material.icons.rounded.Code
|
||||||
|
import androidx.compose.material.icons.rounded.Construction
|
||||||
|
import androidx.compose.material.icons.rounded.ExpandMore
|
||||||
|
import androidx.compose.material.icons.rounded.Public
|
||||||
|
import androidx.compose.material.icons.rounded.Terminal
|
||||||
|
import androidx.compose.material3.Icon
|
||||||
|
import androidx.compose.material3.MaterialTheme
|
||||||
|
import androidx.compose.material3.Text
|
||||||
|
import androidx.compose.runtime.Composable
|
||||||
|
import androidx.compose.runtime.LaunchedEffect
|
||||||
|
import androidx.compose.runtime.getValue
|
||||||
|
import androidx.compose.runtime.mutableLongStateOf
|
||||||
|
import androidx.compose.runtime.mutableStateOf
|
||||||
|
import androidx.compose.runtime.remember
|
||||||
|
import androidx.compose.runtime.setValue
|
||||||
|
import androidx.compose.ui.Alignment
|
||||||
|
import androidx.compose.ui.Modifier
|
||||||
|
import androidx.compose.ui.draw.clip
|
||||||
|
import androidx.compose.ui.draw.rotate
|
||||||
|
import androidx.compose.ui.graphics.vector.ImageVector
|
||||||
|
import androidx.compose.ui.semantics.Role
|
||||||
|
import androidx.compose.ui.semantics.contentDescription
|
||||||
|
import androidx.compose.ui.semantics.role
|
||||||
|
import androidx.compose.ui.semantics.semantics
|
||||||
|
import androidx.compose.ui.semantics.stateDescription
|
||||||
|
import androidx.compose.ui.text.font.FontFamily
|
||||||
|
import androidx.compose.ui.text.font.FontWeight
|
||||||
|
import androidx.compose.ui.text.style.TextOverflow
|
||||||
|
import androidx.compose.ui.tooling.preview.Preview
|
||||||
|
import androidx.compose.ui.unit.Dp
|
||||||
|
import androidx.compose.ui.unit.dp
|
||||||
|
import androidx.compose.ui.unit.sp
|
||||||
|
import com.google.gson.GsonBuilder
|
||||||
|
import com.google.gson.JsonParser
|
||||||
|
import com.rainnya.chat.ui.theme.RainnyaTheme
|
||||||
|
import kotlinx.coroutines.delay
|
||||||
|
import org.json.JSONObject
|
||||||
|
import java.util.Locale
|
||||||
|
|
||||||
|
private const val TICK_MS = 200L
|
||||||
|
|
||||||
|
private data class ToolCallInfo(
|
||||||
|
val id: String?,
|
||||||
|
val name: String,
|
||||||
|
val args: String?,
|
||||||
|
val result: String?,
|
||||||
|
val ts: Long,
|
||||||
|
val finishedTs: Long?,
|
||||||
|
)
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 工具调用卡片(可折叠)。
|
||||||
|
*
|
||||||
|
* 数据来自 `ChatMessage.toolCall`(JSON 字符串),形状:
|
||||||
|
* `{"id","name","args","result","ts","finished_ts"}`。
|
||||||
|
*
|
||||||
|
* - 默认折叠;工具执行中(finished_ts 为空)自动展开。
|
||||||
|
* - 头部:工具图标 +「使用工具 {name}」+ 耗时/进行中 + 展开箭头。
|
||||||
|
* - 展开:Args / Result 的 pretty JSON(等宽、限高可滚动)。
|
||||||
|
*/
|
||||||
|
@Composable
|
||||||
|
fun ToolCallCard(
|
||||||
|
toolCallJson: String,
|
||||||
|
modifier: Modifier = Modifier,
|
||||||
|
initiallyExpanded: Boolean = false,
|
||||||
|
) {
|
||||||
|
val info = remember(toolCallJson) { parseToolCall(toolCallJson) }
|
||||||
|
|
||||||
|
if (info == null) {
|
||||||
|
// 无法解析时退化为原始文本,保证信息不丢
|
||||||
|
Text(
|
||||||
|
text = toolCallJson,
|
||||||
|
style = MaterialTheme.typography.labelSmall,
|
||||||
|
fontFamily = FontFamily.Monospace,
|
||||||
|
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||||
|
modifier = modifier.fillMaxWidth(),
|
||||||
|
)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// 执行中默认展开;消息更新(如 finished_ts 到达)后按默认折叠
|
||||||
|
var expanded by remember(toolCallJson) {
|
||||||
|
mutableStateOf(initiallyExpanded || info.finishedTs == null)
|
||||||
|
}
|
||||||
|
|
||||||
|
val hasTiming = info.ts > 0
|
||||||
|
val running = hasTiming && info.finishedTs == null
|
||||||
|
|
||||||
|
var now by remember { mutableLongStateOf(System.currentTimeMillis()) }
|
||||||
|
LaunchedEffect(info.finishedTs, info.ts) {
|
||||||
|
if (running) {
|
||||||
|
while (true) {
|
||||||
|
now = System.currentTimeMillis()
|
||||||
|
delay(TICK_MS)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
val durationText = when {
|
||||||
|
!hasTiming -> ""
|
||||||
|
running -> formatDuration(now - info.ts)
|
||||||
|
else -> formatDuration((info.finishedTs ?: info.ts) - info.ts)
|
||||||
|
}
|
||||||
|
|
||||||
|
val rotation by animateFloatAsState(
|
||||||
|
targetValue = if (expanded) 180f else 0f,
|
||||||
|
animationSpec = tween(durationMillis = 220, easing = FastOutSlowInEasing),
|
||||||
|
label = "chevronRotation",
|
||||||
|
)
|
||||||
|
|
||||||
|
val shape = RoundedCornerShape(12.dp)
|
||||||
|
|
||||||
|
Column(
|
||||||
|
modifier = modifier
|
||||||
|
.fillMaxWidth()
|
||||||
|
.clip(shape)
|
||||||
|
.background(MaterialTheme.colorScheme.surfaceVariant)
|
||||||
|
.border(1.dp, MaterialTheme.colorScheme.outlineVariant.copy(alpha = 0.5f), shape)
|
||||||
|
.clickable(
|
||||||
|
role = Role.Button,
|
||||||
|
onClick = { expanded = !expanded },
|
||||||
|
)
|
||||||
|
.semantics {
|
||||||
|
contentDescription = "工具调用 ${info.name}"
|
||||||
|
stateDescription = if (expanded) "已展开" else "已折叠"
|
||||||
|
}
|
||||||
|
.padding(vertical = 6.dp),
|
||||||
|
) {
|
||||||
|
Row(
|
||||||
|
modifier = Modifier
|
||||||
|
.fillMaxWidth()
|
||||||
|
.padding(horizontal = 10.dp),
|
||||||
|
verticalAlignment = Alignment.CenterVertically,
|
||||||
|
) {
|
||||||
|
Box(
|
||||||
|
modifier = Modifier
|
||||||
|
.size(26.dp)
|
||||||
|
.clip(RoundedCornerShape(7.dp))
|
||||||
|
.background(MaterialTheme.colorScheme.primary.copy(alpha = 0.1f)),
|
||||||
|
contentAlignment = Alignment.Center,
|
||||||
|
) {
|
||||||
|
Icon(
|
||||||
|
imageVector = toolIcon(info.name),
|
||||||
|
contentDescription = null,
|
||||||
|
tint = MaterialTheme.colorScheme.primary,
|
||||||
|
modifier = Modifier.size(16.dp),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
Spacer(Modifier.width(8.dp))
|
||||||
|
|
||||||
|
Text(
|
||||||
|
text = "使用工具 ${info.name}",
|
||||||
|
style = MaterialTheme.typography.labelLarge,
|
||||||
|
fontWeight = FontWeight.Medium,
|
||||||
|
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||||
|
maxLines = 1,
|
||||||
|
overflow = TextOverflow.Ellipsis,
|
||||||
|
modifier = Modifier.weight(1f),
|
||||||
|
)
|
||||||
|
|
||||||
|
Spacer(Modifier.width(6.dp))
|
||||||
|
|
||||||
|
if (running) {
|
||||||
|
RunningDot()
|
||||||
|
Spacer(Modifier.width(5.dp))
|
||||||
|
}
|
||||||
|
|
||||||
|
if (durationText.isNotEmpty()) {
|
||||||
|
Text(
|
||||||
|
text = if (running) "进行中 · $durationText" else "耗时 $durationText",
|
||||||
|
style = MaterialTheme.typography.labelSmall,
|
||||||
|
color = if (running) {
|
||||||
|
MaterialTheme.colorScheme.primary
|
||||||
|
} else {
|
||||||
|
MaterialTheme.colorScheme.onSurfaceVariant.copy(alpha = 0.7f)
|
||||||
|
},
|
||||||
|
maxLines = 1,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
Spacer(Modifier.width(4.dp))
|
||||||
|
|
||||||
|
Icon(
|
||||||
|
imageVector = Icons.Rounded.ExpandMore,
|
||||||
|
contentDescription = null,
|
||||||
|
tint = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||||
|
modifier = Modifier
|
||||||
|
.size(20.dp)
|
||||||
|
.rotate(rotation),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
AnimatedVisibility(
|
||||||
|
visible = expanded,
|
||||||
|
enter = expandVertically(
|
||||||
|
animationSpec = tween(220, easing = FastOutSlowInEasing),
|
||||||
|
) + fadeIn(tween(160)),
|
||||||
|
exit = shrinkVertically(
|
||||||
|
animationSpec = tween(160, easing = FastOutSlowInEasing),
|
||||||
|
) + fadeOut(tween(100)),
|
||||||
|
) {
|
||||||
|
Column(
|
||||||
|
modifier = Modifier
|
||||||
|
.fillMaxWidth()
|
||||||
|
.padding(top = 6.dp),
|
||||||
|
) {
|
||||||
|
if (info.id != null) {
|
||||||
|
Row(
|
||||||
|
modifier = Modifier
|
||||||
|
.fillMaxWidth()
|
||||||
|
.padding(horizontal = 10.dp, vertical = 2.dp),
|
||||||
|
verticalAlignment = Alignment.CenterVertically,
|
||||||
|
) {
|
||||||
|
Text(
|
||||||
|
text = "ID",
|
||||||
|
style = MaterialTheme.typography.labelSmall,
|
||||||
|
fontWeight = FontWeight.SemiBold,
|
||||||
|
color = MaterialTheme.colorScheme.primary,
|
||||||
|
)
|
||||||
|
Spacer(Modifier.width(8.dp))
|
||||||
|
Text(
|
||||||
|
text = info.id,
|
||||||
|
style = MaterialTheme.typography.labelSmall,
|
||||||
|
fontFamily = FontFamily.Monospace,
|
||||||
|
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||||
|
maxLines = 1,
|
||||||
|
overflow = TextOverflow.Ellipsis,
|
||||||
|
modifier = Modifier.weight(1f),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
JsonDetail(label = "Args", json = prettyJson(info.args), maxHeight = 160.dp)
|
||||||
|
JsonDetail(label = "Result", json = prettyJson(info.result), maxHeight = 240.dp)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@Composable
|
||||||
|
private fun JsonDetail(label: String, json: String?, maxHeight: Dp) {
|
||||||
|
if (json == null) return
|
||||||
|
|
||||||
|
Column(
|
||||||
|
modifier = Modifier
|
||||||
|
.fillMaxWidth()
|
||||||
|
.padding(horizontal = 10.dp, vertical = 3.dp),
|
||||||
|
) {
|
||||||
|
Text(
|
||||||
|
text = label,
|
||||||
|
style = MaterialTheme.typography.labelSmall,
|
||||||
|
fontWeight = FontWeight.SemiBold,
|
||||||
|
color = MaterialTheme.colorScheme.primary,
|
||||||
|
)
|
||||||
|
Spacer(Modifier.height(3.dp))
|
||||||
|
Text(
|
||||||
|
text = json,
|
||||||
|
style = MaterialTheme.typography.bodySmall,
|
||||||
|
fontFamily = FontFamily.Monospace,
|
||||||
|
fontSize = 12.sp,
|
||||||
|
lineHeight = 17.sp,
|
||||||
|
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||||
|
modifier = Modifier
|
||||||
|
.fillMaxWidth()
|
||||||
|
.clip(RoundedCornerShape(8.dp))
|
||||||
|
.background(MaterialTheme.colorScheme.surfaceContainerHighest)
|
||||||
|
.heightIn(max = maxHeight)
|
||||||
|
.verticalScroll(rememberScrollState())
|
||||||
|
.padding(horizontal = 10.dp, vertical = 8.dp),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@Composable
|
||||||
|
private fun RunningDot(modifier: Modifier = Modifier) {
|
||||||
|
val transition = rememberInfiniteTransition(label = "toolRunning")
|
||||||
|
val alpha by transition.animateFloat(
|
||||||
|
initialValue = 0.35f,
|
||||||
|
targetValue = 1f,
|
||||||
|
animationSpec = infiniteRepeatable(tween(600), RepeatMode.Reverse),
|
||||||
|
label = "runningAlpha",
|
||||||
|
)
|
||||||
|
Box(
|
||||||
|
modifier = modifier
|
||||||
|
.size(7.dp)
|
||||||
|
.clip(CircleShape)
|
||||||
|
.background(MaterialTheme.colorScheme.primary.copy(alpha = alpha)),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun toolIcon(name: String): ImageVector {
|
||||||
|
val n = name.lowercase(Locale.ROOT)
|
||||||
|
return when {
|
||||||
|
n.contains("ipython") || n.contains("python") -> Icons.Rounded.Code
|
||||||
|
n.contains("web_search") || n.contains("tavily") -> Icons.Rounded.Public
|
||||||
|
n.contains("shell") || n.contains("console") -> Icons.Rounded.Terminal
|
||||||
|
else -> Icons.Rounded.Construction
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun parseToolCall(json: String): ToolCallInfo? {
|
||||||
|
return try {
|
||||||
|
val obj = JSONObject(json)
|
||||||
|
ToolCallInfo(
|
||||||
|
id = obj.optString("id").takeIf { it.isNotBlank() },
|
||||||
|
name = obj.optString("name").ifBlank { "tool" },
|
||||||
|
args = obj.optString("args").takeIf { it.isNotBlank() },
|
||||||
|
result = if (obj.isNull("result")) null else obj.optString("result").takeIf { it.isNotBlank() },
|
||||||
|
ts = obj.optLong("ts", 0L),
|
||||||
|
finishedTs = if (obj.isNull("finished_ts")) {
|
||||||
|
null
|
||||||
|
} else {
|
||||||
|
obj.optLong("finished_ts", 0L).takeIf { it > 0 }
|
||||||
|
},
|
||||||
|
)
|
||||||
|
} catch (_: Exception) {
|
||||||
|
null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private val prettyPrinter = GsonBuilder().setPrettyPrinting().create()
|
||||||
|
|
||||||
|
/** 可解析则缩进格式化,否则原样返回。 */
|
||||||
|
private fun prettyJson(raw: String?): String? {
|
||||||
|
if (raw.isNullOrBlank()) return null
|
||||||
|
return try {
|
||||||
|
val element = JsonParser.parseString(raw.trim())
|
||||||
|
if (element.isJsonNull) null else prettyPrinter.toJson(element)
|
||||||
|
} catch (_: Exception) {
|
||||||
|
raw
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun formatDuration(ms: Long): String {
|
||||||
|
if (ms < 0) return "0ms"
|
||||||
|
return when {
|
||||||
|
ms < 1_000 -> "${ms}ms"
|
||||||
|
ms < 60_000 -> String.format(Locale.US, "%.1fs", ms / 1000.0)
|
||||||
|
else -> {
|
||||||
|
val minutes = ms / 60_000
|
||||||
|
val secs = (ms % 60_000) / 1_000
|
||||||
|
"${minutes}m ${secs}s"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@Preview(showBackground = true, widthDp = 360)
|
||||||
|
@Composable
|
||||||
|
private fun ToolCallCardRunningPreview() {
|
||||||
|
RainnyaTheme {
|
||||||
|
ToolCallCard(
|
||||||
|
toolCallJson = """{"id":"chatcmpl-tool-r1","name":"astrbot_execute_python","args":"{\"code\":\"import requests\\nr = requests.get('https://api.example.com')\"}","result":null,"ts":1756300000000,"finished_ts":null}""",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@Preview(showBackground = true, widthDp = 360)
|
||||||
|
@Composable
|
||||||
|
private fun ToolCallCardFinishedExpandedPreview() {
|
||||||
|
RainnyaTheme {
|
||||||
|
ToolCallCard(
|
||||||
|
toolCallJson = """{"id":"chatcmpl-tool-f1","name":"astrbot_web_search","args":"{\"query\":\"今天深圳天气怎么样\"}","result":"{\"results\":[{\"title\":\"天气网\",\"snippet\":\"晴 26℃\"},{\"title\":\"预报\",\"snippet\":\"夜间多云\"}]}","ts":1756300000000,"finished_ts":1756300001500}""",
|
||||||
|
initiallyExpanded = true,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,11 +1,17 @@
|
|||||||
package com.rainnya.chat.ui.navigation
|
package com.rainnya.chat.ui.navigation
|
||||||
|
|
||||||
|
import androidx.compose.foundation.layout.Box
|
||||||
|
import androidx.compose.foundation.layout.fillMaxHeight
|
||||||
|
import androidx.compose.foundation.layout.fillMaxSize
|
||||||
import androidx.compose.foundation.layout.padding
|
import androidx.compose.foundation.layout.padding
|
||||||
|
import androidx.compose.foundation.layout.widthIn
|
||||||
import androidx.compose.material.icons.Icons
|
import androidx.compose.material.icons.Icons
|
||||||
import androidx.compose.material.icons.automirrored.filled.Chat
|
import androidx.compose.material.icons.automirrored.filled.Chat
|
||||||
import androidx.compose.material.icons.automirrored.outlined.Chat
|
import androidx.compose.material.icons.automirrored.outlined.Chat
|
||||||
|
import androidx.compose.material.icons.filled.Call
|
||||||
import androidx.compose.material.icons.filled.Forum
|
import androidx.compose.material.icons.filled.Forum
|
||||||
import androidx.compose.material.icons.filled.Settings
|
import androidx.compose.material.icons.filled.Settings
|
||||||
|
import androidx.compose.material.icons.outlined.Call
|
||||||
import androidx.compose.material.icons.outlined.Forum
|
import androidx.compose.material.icons.outlined.Forum
|
||||||
import androidx.compose.material.icons.outlined.Settings
|
import androidx.compose.material.icons.outlined.Settings
|
||||||
import androidx.compose.material3.Icon
|
import androidx.compose.material3.Icon
|
||||||
@@ -18,14 +24,18 @@ import androidx.compose.runtime.getValue
|
|||||||
import androidx.compose.runtime.mutableIntStateOf
|
import androidx.compose.runtime.mutableIntStateOf
|
||||||
import androidx.compose.runtime.saveable.rememberSaveable
|
import androidx.compose.runtime.saveable.rememberSaveable
|
||||||
import androidx.compose.runtime.setValue
|
import androidx.compose.runtime.setValue
|
||||||
|
import androidx.compose.ui.Alignment
|
||||||
import androidx.compose.ui.Modifier
|
import androidx.compose.ui.Modifier
|
||||||
import androidx.compose.ui.graphics.vector.ImageVector
|
import androidx.compose.ui.graphics.vector.ImageVector
|
||||||
|
import androidx.compose.ui.unit.dp
|
||||||
import androidx.lifecycle.viewmodel.compose.viewModel
|
import androidx.lifecycle.viewmodel.compose.viewModel
|
||||||
import com.rainnya.chat.data.settings.AppSettings
|
import com.rainnya.chat.data.settings.AppSettings
|
||||||
import com.rainnya.chat.ui.chat.ChatScreen
|
import com.rainnya.chat.ui.chat.ChatScreen
|
||||||
import com.rainnya.chat.ui.chat.ChatViewModel
|
import com.rainnya.chat.ui.chat.ChatViewModel
|
||||||
import com.rainnya.chat.ui.sessions.SessionsScreen
|
import com.rainnya.chat.ui.sessions.SessionsScreen
|
||||||
import com.rainnya.chat.ui.settings.SettingsScreen
|
import com.rainnya.chat.ui.settings.SettingsScreen
|
||||||
|
import com.rainnya.chat.ui.voice.VoiceCallScreen
|
||||||
|
import com.rainnya.chat.ui.voice.VoiceCallViewModel
|
||||||
|
|
||||||
data class BottomNavItem(
|
data class BottomNavItem(
|
||||||
val label: String,
|
val label: String,
|
||||||
@@ -36,14 +46,40 @@ data class BottomNavItem(
|
|||||||
private val navItems = listOf(
|
private val navItems = listOf(
|
||||||
BottomNavItem("聊天", Icons.AutoMirrored.Filled.Chat, Icons.AutoMirrored.Outlined.Chat),
|
BottomNavItem("聊天", Icons.AutoMirrored.Filled.Chat, Icons.AutoMirrored.Outlined.Chat),
|
||||||
BottomNavItem("会话", Icons.Filled.Forum, Icons.Outlined.Forum),
|
BottomNavItem("会话", Icons.Filled.Forum, Icons.Outlined.Forum),
|
||||||
|
BottomNavItem("通话", Icons.Filled.Call, Icons.Outlined.Call),
|
||||||
BottomNavItem("设置", Icons.Filled.Settings, Icons.Outlined.Settings),
|
BottomNavItem("设置", Icons.Filled.Settings, Icons.Outlined.Settings),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 平板 / 大屏「内容限宽 + 居中」:聊天 / 会话 / 设置统一约束到该最大宽度,
|
||||||
|
* 避免整行文字在大屏上过宽难读。手机宽度天然小于此值 → 约束是 no-op,行为完全不变。
|
||||||
|
* 底部 NavigationBar 保持全宽,不参与限宽。
|
||||||
|
*/
|
||||||
|
private val MainContentMaxWidth = 640.dp
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 限宽容器:占满纵向可用高度(保证键盘 / IME 与 padding 传递不受影响),
|
||||||
|
* 横向最多 MainContentMaxWidth 并由外层 Box 水平居中。
|
||||||
|
*/
|
||||||
|
@Composable
|
||||||
|
private fun MainContentBox(content: @Composable () -> Unit) {
|
||||||
|
Box(
|
||||||
|
modifier = Modifier
|
||||||
|
.fillMaxHeight()
|
||||||
|
.widthIn(max = MainContentMaxWidth),
|
||||||
|
) {
|
||||||
|
content()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
@Composable
|
@Composable
|
||||||
fun AppNavigation(settings: AppSettings) {
|
fun AppNavigation(settings: AppSettings) {
|
||||||
var selectedIndex by rememberSaveable { mutableIntStateOf(0) }
|
var selectedIndex by rememberSaveable { mutableIntStateOf(0) }
|
||||||
val chatViewModel: ChatViewModel = viewModel()
|
val chatViewModel: ChatViewModel = viewModel()
|
||||||
val repository = chatViewModel.repository
|
val repository = chatViewModel.repository
|
||||||
|
val voiceViewModel: VoiceCallViewModel = viewModel(
|
||||||
|
factory = VoiceCallViewModel.factory(repository, settings)
|
||||||
|
)
|
||||||
|
|
||||||
Scaffold(
|
Scaffold(
|
||||||
bottomBar = {
|
bottomBar = {
|
||||||
@@ -64,24 +100,43 @@ fun AppNavigation(settings: AppSettings) {
|
|||||||
}
|
}
|
||||||
},
|
},
|
||||||
) { padding ->
|
) { padding ->
|
||||||
when (selectedIndex) {
|
Box(
|
||||||
0 -> ChatScreen(
|
modifier = Modifier.fillMaxSize(),
|
||||||
scaffoldPadding = padding,
|
contentAlignment = Alignment.TopCenter,
|
||||||
viewModel = chatViewModel,
|
) {
|
||||||
)
|
when (selectedIndex) {
|
||||||
1 -> SessionsScreen(
|
0 -> MainContentBox {
|
||||||
repository = repository,
|
ChatScreen(
|
||||||
onSessionClick = { sessionId ->
|
scaffoldPadding = padding,
|
||||||
chatViewModel.switchSession(sessionId)
|
viewModel = chatViewModel,
|
||||||
selectedIndex = 0
|
)
|
||||||
},
|
}
|
||||||
modifier = Modifier.padding(bottom = padding.calculateBottomPadding()),
|
1 -> MainContentBox {
|
||||||
)
|
SessionsScreen(
|
||||||
2 -> SettingsScreen(
|
repository = repository,
|
||||||
settings = settings,
|
onSessionClick = { sessionId ->
|
||||||
repository = repository,
|
chatViewModel.switchSession(sessionId)
|
||||||
modifier = Modifier.padding(bottom = padding.calculateBottomPadding()),
|
selectedIndex = 0
|
||||||
)
|
},
|
||||||
|
modifier = Modifier.padding(bottom = padding.calculateBottomPadding()),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
// 通话页独立全屏布局:自己的内容限宽(更窄 480dp)在 VoiceCallScreen 内部处理,
|
||||||
|
// 渐变背景保持全屏铺满,故不走统一的 MainContentBox。
|
||||||
|
2 -> VoiceCallScreen(
|
||||||
|
viewModel = voiceViewModel,
|
||||||
|
settings = settings,
|
||||||
|
onOpenSettings = { selectedIndex = 3 },
|
||||||
|
modifier = Modifier.padding(bottom = padding.calculateBottomPadding()),
|
||||||
|
)
|
||||||
|
3 -> MainContentBox {
|
||||||
|
SettingsScreen(
|
||||||
|
settings = settings,
|
||||||
|
repository = repository,
|
||||||
|
modifier = Modifier.padding(bottom = padding.calculateBottomPadding()),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ import androidx.compose.foundation.verticalScroll
|
|||||||
import androidx.compose.material.icons.Icons
|
import androidx.compose.material.icons.Icons
|
||||||
import androidx.compose.material.icons.filled.Key
|
import androidx.compose.material.icons.filled.Key
|
||||||
import androidx.compose.material.icons.filled.Link
|
import androidx.compose.material.icons.filled.Link
|
||||||
|
import androidx.compose.material.icons.filled.Mic
|
||||||
import androidx.compose.material.icons.filled.Person
|
import androidx.compose.material.icons.filled.Person
|
||||||
import androidx.compose.material3.Button
|
import androidx.compose.material3.Button
|
||||||
import androidx.compose.material3.ButtonDefaults
|
import androidx.compose.material3.ButtonDefaults
|
||||||
@@ -55,6 +56,9 @@ fun SettingsScreen(
|
|||||||
var serverUrl by remember { mutableStateOf(settings.serverUrl) }
|
var serverUrl by remember { mutableStateOf(settings.serverUrl) }
|
||||||
var apiKey by remember { mutableStateOf(settings.apiKey) }
|
var apiKey by remember { mutableStateOf(settings.apiKey) }
|
||||||
var username by remember { mutableStateOf(settings.username) }
|
var username by remember { mutableStateOf(settings.username) }
|
||||||
|
var mimoApiKey by remember { mutableStateOf(settings.mimoApiKey) }
|
||||||
|
var mimoBaseUrl by remember { mutableStateOf(settings.mimoBaseUrl) }
|
||||||
|
var ttsVoice by remember { mutableStateOf(settings.ttsVoice) }
|
||||||
var testResult by remember { mutableStateOf<String?>(null) }
|
var testResult by remember { mutableStateOf<String?>(null) }
|
||||||
var testing by remember { mutableStateOf(false) }
|
var testing by remember { mutableStateOf(false) }
|
||||||
val scope = rememberCoroutineScope()
|
val scope = rememberCoroutineScope()
|
||||||
@@ -186,6 +190,77 @@ fun SettingsScreen(
|
|||||||
style = MaterialTheme.typography.bodySmall,
|
style = MaterialTheme.typography.bodySmall,
|
||||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
Spacer(Modifier.height(32.dp))
|
||||||
|
|
||||||
|
Text(
|
||||||
|
text = "语音通话",
|
||||||
|
style = MaterialTheme.typography.titleMedium,
|
||||||
|
color = MaterialTheme.colorScheme.primary,
|
||||||
|
)
|
||||||
|
Spacer(Modifier.height(16.dp))
|
||||||
|
|
||||||
|
OutlinedTextField(
|
||||||
|
value = mimoApiKey,
|
||||||
|
onValueChange = {
|
||||||
|
mimoApiKey = it
|
||||||
|
settings.mimoApiKey = it
|
||||||
|
},
|
||||||
|
label = { Text("MiMo API Key(可选)") },
|
||||||
|
placeholder = { Text("sk-…") },
|
||||||
|
leadingIcon = { Icon(Icons.Default.Key, contentDescription = null) },
|
||||||
|
singleLine = true,
|
||||||
|
visualTransformation = PasswordVisualTransformation(),
|
||||||
|
keyboardOptions = KeyboardOptions(imeAction = ImeAction.Next),
|
||||||
|
modifier = Modifier.fillMaxWidth(),
|
||||||
|
)
|
||||||
|
|
||||||
|
Spacer(Modifier.height(4.dp))
|
||||||
|
Text(
|
||||||
|
text = "留空则自动使用服务器已配置的语音",
|
||||||
|
style = MaterialTheme.typography.bodySmall,
|
||||||
|
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||||
|
)
|
||||||
|
|
||||||
|
Spacer(Modifier.height(12.dp))
|
||||||
|
|
||||||
|
OutlinedTextField(
|
||||||
|
value = mimoBaseUrl,
|
||||||
|
onValueChange = {
|
||||||
|
mimoBaseUrl = it
|
||||||
|
settings.mimoBaseUrl = it
|
||||||
|
},
|
||||||
|
label = { Text("服务器地址") },
|
||||||
|
placeholder = { Text("https://api.xiaomimimo.com/v1") },
|
||||||
|
leadingIcon = { Icon(Icons.Default.Link, contentDescription = null) },
|
||||||
|
singleLine = true,
|
||||||
|
keyboardOptions = KeyboardOptions(imeAction = ImeAction.Next),
|
||||||
|
modifier = Modifier.fillMaxWidth(),
|
||||||
|
)
|
||||||
|
|
||||||
|
Spacer(Modifier.height(12.dp))
|
||||||
|
|
||||||
|
OutlinedTextField(
|
||||||
|
value = ttsVoice,
|
||||||
|
onValueChange = {
|
||||||
|
ttsVoice = it
|
||||||
|
settings.ttsVoice = it
|
||||||
|
},
|
||||||
|
label = { Text("音色") },
|
||||||
|
placeholder = { Text("冰糖") },
|
||||||
|
leadingIcon = { Icon(Icons.Default.Mic, contentDescription = null) },
|
||||||
|
singleLine = true,
|
||||||
|
keyboardOptions = KeyboardOptions(imeAction = ImeAction.Done),
|
||||||
|
modifier = Modifier.fillMaxWidth(),
|
||||||
|
)
|
||||||
|
|
||||||
|
Spacer(Modifier.height(8.dp))
|
||||||
|
|
||||||
|
Text(
|
||||||
|
text = "语音内容会上传小米 MiMo 云端识别,请勿提及敏感信息",
|
||||||
|
style = MaterialTheme.typography.bodySmall,
|
||||||
|
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||||
|
)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,163 @@
|
|||||||
|
package com.rainnya.chat.ui.voice
|
||||||
|
|
||||||
|
import android.Manifest
|
||||||
|
import android.app.Application
|
||||||
|
import android.content.pm.PackageManager
|
||||||
|
import androidx.core.content.ContextCompat
|
||||||
|
import androidx.lifecycle.AndroidViewModel
|
||||||
|
import androidx.lifecycle.ViewModel
|
||||||
|
import androidx.lifecycle.ViewModelProvider
|
||||||
|
import androidx.lifecycle.viewModelScope
|
||||||
|
import androidx.lifecycle.viewmodel.CreationExtras
|
||||||
|
import com.rainnya.chat.data.repository.ChatRepository
|
||||||
|
import com.rainnya.chat.data.repository.ConnectionState
|
||||||
|
import com.rainnya.chat.data.settings.AppSettings
|
||||||
|
import com.rainnya.chat.data.voice.CallState
|
||||||
|
import com.rainnya.chat.data.voice.ProviderFetcher
|
||||||
|
import com.rainnya.chat.data.voice.VoiceCallEngine
|
||||||
|
import com.rainnya.chat.data.voice.VoiceConfigState
|
||||||
|
import kotlinx.coroutines.flow.MutableStateFlow
|
||||||
|
import kotlinx.coroutines.flow.StateFlow
|
||||||
|
import kotlinx.coroutines.launch
|
||||||
|
|
||||||
|
data class VoiceCallUiState(
|
||||||
|
val callState: CallState = CallState.Idle,
|
||||||
|
val statusText: String = "空闲",
|
||||||
|
val transcribedText: String? = null,
|
||||||
|
val error: String? = null,
|
||||||
|
val micPermissionGranted: Boolean = false,
|
||||||
|
)
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 语音通话页 ViewModel:把 VoiceCallEngine 的 state/statusText/transcribedText/lastError
|
||||||
|
* 映射到 VoiceCallUiState,并负责把麦克风权限结果透传给引擎。
|
||||||
|
*/
|
||||||
|
class VoiceCallViewModel(
|
||||||
|
application: Application,
|
||||||
|
private val repository: ChatRepository,
|
||||||
|
private val settings: AppSettings,
|
||||||
|
) : AndroidViewModel(application) {
|
||||||
|
|
||||||
|
private val engine = VoiceCallEngine(viewModelScope, settings, repository)
|
||||||
|
|
||||||
|
/** 从 AstrBot 服务器自动拉取语音配置(复用服务端已配好的 MiMo key) */
|
||||||
|
private val fetcher = ProviderFetcher(settings)
|
||||||
|
|
||||||
|
private val _uiState = MutableStateFlow(
|
||||||
|
VoiceCallUiState(micPermissionGranted = checkMicPermission()),
|
||||||
|
)
|
||||||
|
val uiState: StateFlow<VoiceCallUiState> = _uiState
|
||||||
|
|
||||||
|
/** 服务器语音配置拉取状态(初始 Fetching,供 UI 绑定) */
|
||||||
|
private val _configState = MutableStateFlow(VoiceConfigState.Fetching)
|
||||||
|
val configState: StateFlow<VoiceConfigState> = _configState
|
||||||
|
|
||||||
|
/** 配置拉取失败原因(configState == FetchFailed 时有值),供 UI 提示 */
|
||||||
|
private val _configError = MutableStateFlow<String?>(null)
|
||||||
|
val configError: StateFlow<String?> = _configError
|
||||||
|
|
||||||
|
/** 服务器连接状态(转发 repository,供语音页顶部状态条收集,H1-UI) */
|
||||||
|
val connectionState: StateFlow<ConnectionState> = repository.connectionState
|
||||||
|
|
||||||
|
/** 重连服务器(H1-UI) */
|
||||||
|
fun reconnect() = repository.reconnect()
|
||||||
|
|
||||||
|
init {
|
||||||
|
// 已在运行期授权过 → 直接同步给引擎,避免再次弹窗后仍提示"需要权限"
|
||||||
|
if (checkMicPermission()) {
|
||||||
|
engine.onMicPermissionGranted()
|
||||||
|
}
|
||||||
|
// 语音配置:手动已配置 → Ready;未配置 → 尝试从服务器自动拉取
|
||||||
|
if (settings.isVoiceConfigured) {
|
||||||
|
_configState.value = VoiceConfigState.Ready
|
||||||
|
} else {
|
||||||
|
refreshConfig()
|
||||||
|
}
|
||||||
|
viewModelScope.launch {
|
||||||
|
engine.state.collect { state ->
|
||||||
|
_uiState.value = _uiState.value.copy(callState = state)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
viewModelScope.launch {
|
||||||
|
engine.statusText.collect { text ->
|
||||||
|
_uiState.value = _uiState.value.copy(statusText = text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
viewModelScope.launch {
|
||||||
|
engine.transcribedText.collect { text ->
|
||||||
|
_uiState.value = _uiState.value.copy(transcribedText = text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
viewModelScope.launch {
|
||||||
|
engine.lastError.collect { error ->
|
||||||
|
_uiState.value = _uiState.value.copy(error = error)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fun startTalking() {
|
||||||
|
_uiState.value = _uiState.value.copy(error = null)
|
||||||
|
engine.startTalking()
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 手动触发从服务器拉取语音配置(供 UI「重新拉取/重试」按钮调用) */
|
||||||
|
fun refreshConfig() {
|
||||||
|
_configState.value = VoiceConfigState.Fetching
|
||||||
|
_configError.value = null
|
||||||
|
viewModelScope.launch {
|
||||||
|
val state = fetcher.fetchServerVoiceConfig()
|
||||||
|
_configState.value = state
|
||||||
|
if (state == VoiceConfigState.FetchFailed) {
|
||||||
|
_configError.value =
|
||||||
|
fetcher.lastFailureReason ?: "从服务器拉取语音配置失败,请检查服务器或稍后重试"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fun stopTalking() = engine.stopTalking()
|
||||||
|
|
||||||
|
fun hangUp() {
|
||||||
|
_uiState.value = _uiState.value.copy(error = null)
|
||||||
|
engine.hangUp()
|
||||||
|
}
|
||||||
|
|
||||||
|
fun onPermissionResult(granted: Boolean) {
|
||||||
|
if (granted) {
|
||||||
|
engine.onMicPermissionGranted()
|
||||||
|
_uiState.value = _uiState.value.copy(micPermissionGranted = true, error = null)
|
||||||
|
} else {
|
||||||
|
_uiState.value = _uiState.value.copy(
|
||||||
|
micPermissionGranted = false,
|
||||||
|
error = "权限被拒,无法开始通话",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun checkMicPermission(): Boolean =
|
||||||
|
ContextCompat.checkSelfPermission(
|
||||||
|
getApplication(),
|
||||||
|
Manifest.permission.RECORD_AUDIO,
|
||||||
|
) == PackageManager.PERMISSION_GRANTED
|
||||||
|
|
||||||
|
companion object {
|
||||||
|
/**
|
||||||
|
* 供 UI 通过 `viewModel(factory = VoiceCallViewModel.factory(repository, settings))` 使用。
|
||||||
|
* Application 从 CreationExtras 中取,无需重复注入。
|
||||||
|
*/
|
||||||
|
fun factory(repository: ChatRepository, settings: AppSettings): ViewModelProvider.Factory =
|
||||||
|
object : ViewModelProvider.Factory {
|
||||||
|
@Suppress("UNCHECKED_CAST")
|
||||||
|
override fun <T : ViewModel> create(modelClass: Class<T>, extras: CreationExtras): T {
|
||||||
|
val app = extras[ViewModelProvider.AndroidViewModelFactory.APPLICATION_KEY]
|
||||||
|
?: throw IllegalStateException("缺少 Application,请通过 viewModel(factory = ...) 创建")
|
||||||
|
return VoiceCallViewModel(app, repository, settings) as T
|
||||||
|
}
|
||||||
|
|
||||||
|
override fun <T : ViewModel> create(modelClass: Class<T>): T {
|
||||||
|
throw UnsupportedOperationException(
|
||||||
|
"请通过 compose 的 viewModel(factory = VoiceCallViewModel.factory(...)) 创建以提供 Application",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,13 +1,8 @@
|
|||||||
<?xml version="1.0" encoding="utf-8"?><!--
|
<?xml version="1.0" encoding="utf-8"?><!--
|
||||||
Sample backup rules file; uncomment and customize as necessary.
|
Backup rules — 排除 SharedPreferences,避免 API Key / MiMo Key 等敏感配置进云端备份(M3)。
|
||||||
See https://developer.android.com/guide/topics/data/autobackup
|
|
||||||
for details.
|
|
||||||
Note: This file is ignored for devices older than API 31
|
Note: This file is ignored for devices older than API 31
|
||||||
See https://developer.android.com/about/versions/12/backup-restore
|
See https://developer.android.com/about/versions/12/backup-restore
|
||||||
-->
|
-->
|
||||||
<full-backup-content>
|
<full-backup-content>
|
||||||
<!--
|
<exclude domain="sharedpref" path="rainnya_prefs.xml"/>
|
||||||
<include domain="sharedpref" path="."/>
|
|
||||||
<exclude domain="sharedpref" path="device.xml"/>
|
|
||||||
-->
|
|
||||||
</full-backup-content>
|
</full-backup-content>
|
||||||
@@ -1,19 +1,13 @@
|
|||||||
<?xml version="1.0" encoding="utf-8"?><!--
|
<?xml version="1.0" encoding="utf-8"?><!--
|
||||||
Sample data extraction rules file; uncomment and customize as necessary.
|
Data extraction rules — 云备份与设备迁移均排除 SharedPreferences,
|
||||||
|
避免 api_key / mimo_api_key 等敏感配置被导出(M3)。
|
||||||
See https://developer.android.com/about/versions/12/backup-restore#xml-changes
|
See https://developer.android.com/about/versions/12/backup-restore#xml-changes
|
||||||
for details.
|
|
||||||
-->
|
-->
|
||||||
<data-extraction-rules>
|
<data-extraction-rules>
|
||||||
<cloud-backup>
|
<cloud-backup>
|
||||||
<!-- TODO: Use <include> and <exclude> to control what is backed up.
|
<exclude domain="sharedpref" path="rainnya_prefs.xml"/>
|
||||||
<include .../>
|
|
||||||
<exclude .../>
|
|
||||||
-->
|
|
||||||
</cloud-backup>
|
</cloud-backup>
|
||||||
<!--
|
|
||||||
<device-transfer>
|
<device-transfer>
|
||||||
<include .../>
|
<exclude domain="sharedpref" path="rainnya_prefs.xml"/>
|
||||||
<exclude .../>
|
|
||||||
</device-transfer>
|
</device-transfer>
|
||||||
-->
|
|
||||||
</data-extraction-rules>
|
</data-extraction-rules>
|
||||||
@@ -1,4 +1,20 @@
|
|||||||
<?xml version="1.0" encoding="utf-8"?>
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<!--
|
||||||
|
Cleartext 策略(L4):
|
||||||
|
- AstrBot 默认走内网 http://192.168.1.100:6185,需要放行 cleartext;
|
||||||
|
MiMo ASR/TTS 走 https(api.xiaomimimo.com),不受本配置影响。
|
||||||
|
- 理想做法是把 cleartext 只放行给 AstrBot 内网 IP 的 domain-config,并把 base-config
|
||||||
|
改为 false。但 Android Network Security Config 的 <domain> 仅支持域名匹配,
|
||||||
|
**不支持 IP 字面量**(192.168.1.100 不是有效 domain),实测会导致内网 AstrBot 连接被拦。
|
||||||
|
故暂保留 base-config cleartextTrafficPermitted="true" 兜底(风险:任意 http 明文流量
|
||||||
|
均被放行,仅限内网/测试环境使用)。
|
||||||
|
- 若日后改用内网域名(如 astrobot.local),可启用下方注释块并关闭 base-config:
|
||||||
|
-->
|
||||||
<network-security-config>
|
<network-security-config>
|
||||||
<base-config cleartextTrafficPermitted="true" />
|
<base-config cleartextTrafficPermitted="true" />
|
||||||
|
<!--
|
||||||
|
<domain-config cleartextTrafficPermitted="true">
|
||||||
|
<domain includeSubdomains="true">astrobot.local</domain>
|
||||||
|
</domain-config>
|
||||||
|
-->
|
||||||
</network-security-config>
|
</network-security-config>
|
||||||
|
|||||||
@@ -0,0 +1,294 @@
|
|||||||
|
# 雨喵语音通话(豆包式)实现方案
|
||||||
|
|
||||||
|
> 状态:实现方案(基于 2026-08-22 设计文档《语音通话方案.md》的评审 + MiMo API 核验 + 架构评审 + 红队审查修订 + 完整实现计划)
|
||||||
|
> 范围:rainnya-chat (Android, Kotlin/Compose) 端实现,**AstrBot 服务端零改动**
|
||||||
|
> 版本:v1.1 · 2026-08-23(含红队审查修订,见 §10)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. 结论速览(TL;DR)
|
||||||
|
|
||||||
|
- **方向正确,采纳并唯一化**:App 直连 MiMo ASR/TTS + 走现有 WS 给 AstrBot LLM。服务端 STT/TTS 备选路线在 AstrBot 代码层面走不通(openapi WS 纯文本 + 原生 TTS 无流式实现),**降级为未来演进方向**。
|
||||||
|
- 全部复杂度收敛进独立组件 `data/voice/VoiceCallEngine`(观察者模式),ChatRepository 只加一个 `sendVoiceMessage(text)`,Room **零 schema 变更**。
|
||||||
|
- 端到端目标:**说完 → 雨喵开口 < 2.5s**(理想 < 2.0s),需"边等 LLM 边流式 TTS 边播"+ 句级分段。
|
||||||
|
- 最优先验证:**PCM 格式实测 spike**(决定 TTS→AudioTrack 整条链路成败)。
|
||||||
|
- 红队审查已做:管线边界语义(打断原子操作 / end 冲刷 / 句序重排 / wsEvents 驱动)、前台强制机制、key 备份防护、YAGNI 砍项——详见 §3.6 与 §10。
|
||||||
|
- 工期:单人约 **2.5~3 周**(M0~M5)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 设计文档评审
|
||||||
|
|
||||||
|
### 1.1 总体评价
|
||||||
|
|
||||||
|
设计文档主线正确、API 细节绝大部分准确、当前(2026-08)依然有效。三条关键结论:
|
||||||
|
|
||||||
|
1. 方案采用 **App 客户端直连 MiMo**(ASR/TTS)而非服务端——正确,且经源码核实这是**唯一可行路线**。
|
||||||
|
2. 非直观要求(TTS 文字放 assistant 消息)**属实且是硬性要求**。
|
||||||
|
3. 方案缺一个关键取舍说明:**AstrBot 原生 TTS 是非流式的**,低延迟流式(可打断)只有客户端直连才有。
|
||||||
|
|
||||||
|
### 1.2 API 核验结果(官方文档 + AstrBot v4.27.2 源码交叉验证,2026-08)
|
||||||
|
|
||||||
|
**准确项(全部通过)**:
|
||||||
|
|
||||||
|
| 声明 | 依据 |
|
||||||
|
|---|---|
|
||||||
|
| ASR 端点 `/v1/chat/completions`(OpenAI 兼容) | 官方 curl 原文 |
|
||||||
|
| `input_audio` 结构 + `data:audio/wav;base64,` data URL | 官方请求体 |
|
||||||
|
| `asr_options.language:"zh"`(更快更准,可选项 auto/zh/en) | 官方文档 |
|
||||||
|
| ASR 只支持 wav/mp3 | 官方原文 |
|
||||||
|
| base64 **编码后** ≤10MB | 官方原文 |
|
||||||
|
| ASR 价格 ¥0.5/音频小时(按输入时长计费) | 官方价格页 |
|
||||||
|
| TTS 端点是 `/v1/chat/completions` 而非 `/audio/speech` | 官方 curl |
|
||||||
|
| TTS 文字必须放 **assistant** 消息(user 只用于风格指令) | 官方硬性要求原文 |
|
||||||
|
| `audio:{format:"pcm16", voice:"冰糖"}` + `stream:true` SSE | 官方文档(低延迟流式已恢复) |
|
||||||
|
| 流式块在 `choices[0].delta.audio.data`,**24kHz PCM16LE mono** | 官方代码注释原文 |
|
||||||
|
| 音色 `冰糖`/`茉莉` 有效(内置表 8 个音色) | 官方音色表 |
|
||||||
|
| TTS 限时免费 | 官方价格页 |
|
||||||
|
| `sk-` 按量付费 key、base URL `https://api.xiaomimimo.com/v1` | 官方 first-api-call |
|
||||||
|
| 参考仓库 `timyang2005/MiMoTTSReader` 存在(纯 Kotlin,OkHttp+Gson) | GitHub API |
|
||||||
|
|
||||||
|
**需修正 / 补充**:
|
||||||
|
|
||||||
|
| # | 位置 | 问题 | 正确内容 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | §八 TTS 文档链接 | 失效(SPA 兜底页) | API 参考 `https://mimo.mi.com/docs/zh-CN/api/audio/tts`;使用指南 `.../quick-start/usage-guide/audio/speech-synthesis-v2.5` |
|
||||||
|
| 2 | §五 stt_health_check.wav | 参数写错 | 实际 **24kHz / 16bit / 单声道 / 3.2s**(`/home/miaomiao/Project/astrbot/samples/stt_health_check.wav`),非"16k 1秒" |
|
||||||
|
| 3 | §二 "WAV 16k 单声道" | 官方未规定采样率 | 仅推荐值(AstrBot 自带健康检查用 24k),标注无官方出处 |
|
||||||
|
| 4 | 建议补充 | 关键取舍遗漏 | AstrBot 原生 TTS 非流式(`mimo_tts_api_source.py` 单次 POST 整块 WAV);SSE pcm16 流式只能客户端直连 |
|
||||||
|
|
||||||
|
> 附带确认:`mimo-v2.5-asr` 是专用 ASR 模型(MiMo-V2 系列 2026-06-30 已下线);AstrBot 原生 STT 走 auto 语言检测(不指定 zh),客户端直连可指定 zh 更准。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 架构决策
|
||||||
|
|
||||||
|
### 2.1 路线取舍:App 直连 MiMo vs AstrBot 服务端 STT/TTS
|
||||||
|
|
||||||
|
| 维度 | App 直连 MiMo(**推荐,唯一路线**) | AstrBot 服务端 STT/TTS(降级为未来方向) |
|
||||||
|
|---|---|---|
|
||||||
|
| 实现复杂度 | **零服务端改动**,纯 App 功能 | 大改:openapi WS 扩展 + 管线 STT 注入 + 流式 TTS 回传 + 新 WS 消息类型 |
|
||||||
|
| 延迟 | ASR/TTS 各一次直连;若 AstrBot 在 rainserver1 远程,客户端直连反而更近 | 音频上行 + 服务端 ASR + 下行 + TTS,多一跳,RTT 翻倍 |
|
||||||
|
| 流式 TTS | ✅ SSE pcm16 流式,可打断 | ❌ 原生 TTS 非流式(`get_audio` 整块),Live Agent 退回 `_simulated_stream_tts` 按句分块,延迟线性爆炸 |
|
||||||
|
| 人格保留 | ✅ 相同(TTS 合成 App 收到的同一段流式文本) | 相同,且可合成工具结果等 App 收不到的中间内容 |
|
||||||
|
| 打断 | 纯客户端(停 AudioTrack + 重录),零协调 | 客户端仍要停播放,服务端已花配额不可回收 |
|
||||||
|
| 网络要求 | App 需直连外网 api.xiaomimimo.com | App 只连内网,外网服务端出 |
|
||||||
|
| 部署 | 发 App 新版本即可 | AstrBot 升级 + App 升级同步 |
|
||||||
|
|
||||||
|
**结论**:采用 App 直连主线,作为唯一路线推进。
|
||||||
|
|
||||||
|
**代价与对策**:
|
||||||
|
- MiMo key 随 APK 分发 → 对策:key 用户自填(SharedPreferences,明文内存态)+ 强制 HTTPS(收窄 `network_security_config` 的 cleartext 只放 AstrBot 内网 IP)+ 预留 `MiMoClient` 抽象以便未来演进"App→服务端代理 MiMo"。
|
||||||
|
- 数据流量上行(16k mono ≈1.9MB/min)、下行(24k ≈2.9MB/min)→ v1 可接受,文档注明。
|
||||||
|
|
||||||
|
### 2.2 核心设计原则
|
||||||
|
|
||||||
|
1. **VoiceCallEngine 是观察者,不是 ChatRepository 的改版**:订阅 `repository.messages`/`wsEvents` 驱动 TTS,调用 `repository.sendVoiceMessage(text)` 注入语音文本。状态机独立(CallState),与 WS/消息状态机解耦。
|
||||||
|
2. **语音只是传输,文字气泡是唯一事实源**:通话内容以文字气泡进会话记录,Room 零变更,天然正确。
|
||||||
|
3. **流式管线 + 句级分段**:不等整段回复,LLM 流式文本按句切分后逐句 TTS。
|
||||||
|
4. **打断 = 停音频 + 保留文本**:LLM 流式文本继续进气泡(特性,保真)。
|
||||||
|
5. **模块可单独测试、单独降级**:TTS 挂了回退文字气泡;ASR 挂了提示重录。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 详细设计
|
||||||
|
|
||||||
|
### 3.1 数据层 `data/voice/`
|
||||||
|
|
||||||
|
```kotlin
|
||||||
|
sealed interface CallState {
|
||||||
|
object Idle; object Recording; object Asr; object WaitingReply
|
||||||
|
object Playing; object Interrupted
|
||||||
|
}
|
||||||
|
|
||||||
|
class VoiceCallEngine( // appContext 注入,生命周期 = Application
|
||||||
|
private val scope: CoroutineScope,
|
||||||
|
private val settings: AppSettings,
|
||||||
|
private val repository: ChatRepository, // 观察,不持有
|
||||||
|
) {
|
||||||
|
val state: StateFlow<CallState>
|
||||||
|
fun startCall(sessionId: String?) // 请求麦克风
|
||||||
|
fun stopCall() // 停引擎;不断 WS、不删会话
|
||||||
|
// 驱动源:只订阅 wsEvents(增量 plain/end 事件)+ 按 message_id 去重,
|
||||||
|
// 绝不订阅 repository.messages 全量列表(切会话/迁移整表替换 → 重复合成历史)
|
||||||
|
// 内部:AudioRecorder(VAD) / MiMoAsrClient / SentenceSplitter /
|
||||||
|
// MiMoTtsClient(SSE) / AudioPlaybackEngine
|
||||||
|
}
|
||||||
|
|
||||||
|
class MiMoAsrClient(baseUrl, apiKey) {
|
||||||
|
suspend fun transcribe(wav: ByteArray): String
|
||||||
|
// POST {base}/v1/chat/completions,model=mimo-v2.5-asr,
|
||||||
|
// input_audio data URL,asr_options.language=zh,编码后≤10MB
|
||||||
|
}
|
||||||
|
|
||||||
|
class MiMoTtsClient(baseUrl, apiKey) {
|
||||||
|
fun synthesizeStream(text: String): Flow<ByteArray>
|
||||||
|
// SSE,model=mimo-v2.5-tts,assistant 放文字,audio{pcm16,voice},
|
||||||
|
// 鉴权:Authorization: Bearer <key>(官方文档 + 本机 AstrBot mimo_api_common 一致;
|
||||||
|
// 参考仓库用 api-key: 头——最终以 M3 spike 真实请求定案)
|
||||||
|
// OkHttp readTimeout(0):SSE 长句合成可能 60s+ 无数据,防中途断流
|
||||||
|
}
|
||||||
|
|
||||||
|
class SentenceSplitter {
|
||||||
|
fun push(token: String): List<String> // 。!?;+ ≤60字硬切,完整句出队
|
||||||
|
fun clear() // 打断/切会话时清残留半句(防并入下一轮)
|
||||||
|
fun flush() // turn 边界(WS end / streaming=false)强制冲刷收尾句
|
||||||
|
}
|
||||||
|
|
||||||
|
class AudioPlaybackEngine(sampleRate = 24000) {
|
||||||
|
fun enqueue(seq: Int, pcm: ByteArray) // 每块带序号,按 seq 重排防句子乱序
|
||||||
|
fun play() // 预滚 150~200ms 后 play()
|
||||||
|
fun stop() // flush 队列 + pause + flush,为打断准备,非 destroy
|
||||||
|
fun release()
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**AudioTrack 关键参数**(minSdk 28 全可用):
|
||||||
|
- `AudioTrack.Builder().setAudioFormat(24kHz/MONO/PCM_16BIT).setBufferSizeInBytes(max(getMinBufferSize, 24000*2*0.5))`(按 300ms 设)
|
||||||
|
- MODE_STREAM + 阻塞 ByteArray 队列 + 后台写线程;`write(byte[],0,n,WRITE_BLOCKING)` 天然背压
|
||||||
|
- 预滚 150~200ms(≈7~9.6KB)吞 SSE 抖动,不过度缓冲
|
||||||
|
- 抗卡顿:预滚耗尽允许 1~2 次短暂静音,不无限等;SSE 断流 → 停止 + 角标 + 文字气泡兜底(可系统 TTS 重读/重试整句)
|
||||||
|
|
||||||
|
### 3.2 配置(AppSettings + 设置页)
|
||||||
|
|
||||||
|
- `AppSettings` 新增 `voiceCall` 配置组:`mimoApiKey / mimoBaseUrl(默认 https://api.xiaomimimo.com/v1) / asrModel(默认 mimo-v2.5-asr) / ttsModel(默认 mimo-v2.5-tts) / ttsVoice(默认 冰糖) / ttsFormat(默认 pcm16)`
|
||||||
|
- 设置页新增"语音通话"区块;key 由用户自填
|
||||||
|
|
||||||
|
### 3.3 UI
|
||||||
|
|
||||||
|
- 聊天页加"通话"入口 → 全屏 `VoiceCallScreen`(Compose 路由),与 ChatScreen 共用 repository 和 sessionId
|
||||||
|
- 通话模式发的消息带同一 `session_id` → 多轮上下文延续,服务端无感知
|
||||||
|
- 最小交互控件集:挂断 / 静音(暂停录音)/ 扬声器切换;状态提示:录音中/识别中/等待回复/播放中/打断
|
||||||
|
- **前台强制机制**(隐私关键):`onStop/onPause` → `engine.pauseRecording()`(保留播放),`onStart/onResume` 恢复;防止切后台后录音继续、误把真人对话发给雨喵
|
||||||
|
- 错误提示:没听清/网络异常/权限拒绝;设置页注明"语音将上传小米 MiMo 云端识别"
|
||||||
|
- 退出通话:`stopCall()` 只停引擎,WS 与会话保持,回聊天页继续文字交流
|
||||||
|
|
||||||
|
### 3.4 ChatRepository 扩展
|
||||||
|
|
||||||
|
```kotlin
|
||||||
|
fun sendVoiceMessage(transcribedText: String) // 内部走 sendMessage 文本路径
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.5 权限与网络
|
||||||
|
|
||||||
|
- **Manifest 新增 `RECORD_AUDIO`**(当前只有 INTERNET)+ 运行时请求 + 拒绝引导 + 录音设备被占用的报错
|
||||||
|
- `network_security_config.xml` 收窄:`cleartextTrafficPermitted` 只放 AstrBot 内网 IP(如 `192.168.1.100` / rainserver),MiMo 强制 TLS
|
||||||
|
|
||||||
|
### 3.6 管线时序与边界语义(红队审查定稿,M3 必须实现)
|
||||||
|
|
||||||
|
1. **打断 = 原子操作(四件事)**:`engine.stop()`(清 PCM 队列)+ `splitter.clear()`(清残留半句)+ 取消在途 SSE 请求(`call.cancel()`)+ 本轮回复"不再合成语音"标记。缺一不可——否则上一轮残留半句会与新一轮开头拼成"缝合句"送 TTS。
|
||||||
|
2. **turn 边界冲刷**:LLM 回复普遍无标点结尾("很高兴认识你"),靠标点永远收不了尾句 → 以 WS `end` 事件 / `streaming=false` 为 turn 边界,`splitter.flush()` 强制冲刷。**依赖 AstrBot 发送 `end` 事件**(行为依赖,零代码改动但必须 pin 住,纳入 §7 集成测试)。
|
||||||
|
3. **句序与防句间断音**:单句 TTS 严格串行会有 ~0.5~1s 句间请求空档 → AudioTrack 断音;并行发射则可能乱序。采用**提前 1~2 句重叠管线 + PCM 块带 seq 序号按序重排**;v1 可降级为有界并行度(如 2)防止乱序过远。
|
||||||
|
4. **驱动源**:引擎只订阅 `wsEvents`(增量 plain/end)+ 按 `message_id` 去重,**不订阅 repository.messages 全量列表**(切会话/迁移整表替换会触发历史重复合成)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 延迟预算与管线
|
||||||
|
|
||||||
|
**目标**:说完 → 雨喵开口 **< 2.5s**(v1 达标线),**< 2.0s** 理想。豆包典型 1.5~2.5s。
|
||||||
|
|
||||||
|
| 阶段 | 可重叠 | 预算 |
|
||||||
|
|---|---|---|
|
||||||
|
| VAD 静音尾 | 否 | 0.5~0.7s(600ms 静音阈值) |
|
||||||
|
| ASR(整句上传+同步 chat/completions) | 否 | 0.4~0.8s |
|
||||||
|
| LLM 首句文本(TTFT+句切) | 仅首句暴露 | 0.8~1.5s |
|
||||||
|
| TTS 首包(SSE 首块) | 是 | 0.3~0.6s |
|
||||||
|
| **暴露给用户合计** | | **~2.0~2.6s** |
|
||||||
|
|
||||||
|
**管线**:
|
||||||
|
```
|
||||||
|
WS plain 流式文本 ─▶ SentenceSplitter(。!?;+≤60字硬切) ─▶ 每完成一句
|
||||||
|
→ MiMo TTS(单句 stateless) ─▶ PCM 队列 ─▶ AudioTrack 顺序播放
|
||||||
|
```
|
||||||
|
|
||||||
|
**VAD 策略(v1)**:能量(RMS) + 静音 600ms;**阈值 = max(固定下限, 前 500ms 噪声均值 × 系数)**(避免动态无下限、耳语永不达标);录音 45s 硬切防成本失控。**"未检测到语音"分支:45s 内从未超过阈值则丢弃本次录音、不发 ASR**(防噪音段白花计费 + 乱码文本)。不上 VAD 模型。
|
||||||
|
|
||||||
|
**两个必须先实测验证的未知数(M3 开头 0.5d spike)**:
|
||||||
|
1. MiMo TTS 是否支持流式/部分文本输入——不支持则用"每句一请求"退化方案(句间损失跨句韵律,v1 够用)
|
||||||
|
2. **返回 PCM 真实采样率/声道/字节序**(首包前 12 字节/抓包确认;AudioTrack 配错=全断音)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 实现里程碑
|
||||||
|
|
||||||
|
| 阶段 | 目标 | 任务 | 验收 | 复杂度 |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| **M0 录音+VAD**(1~2d) | AudioRecord 录 WAV 16k mono + RMS 能量 + 600ms 静音端点检测,45s 硬切 | RECORD_AUDIO 权限;AudioRecorder 组件 | 录一段本地可播放;静音自动停 | 低 |
|
||||||
|
| **M1 MiMo ASR**(2~3d) | base64→`/v1/chat/completions`→文本;配置字段;错误处理 | MiMoAsrClient;AppSettings 组 | 录→ASR 文本正确;空文本/HTTP 错误有提示;未检测到语音→不发 ASR | 低 |
|
||||||
|
| **M2 通话模式+接 WS**(2~3d) | 通话页 UI + `sendVoiceMessage` 走现有 WS,同 session_id | VoiceCallScreen;入口;路由;前台强制(onPause→pauseRecording) | 说完话聊天页出现用户/AI 文本回复,上下文延续;切后台录音暂停 | 中 |
|
||||||
|
| **M3 TTS流式+AudioTrack**(3~4d) | **先 PCM spike**(含鉴权头定案)→ SSE 解析 + 句级分段 + 播放引擎 + **管线正确性(§3.6)** | MiMoTtsClient;SentenceSplitter;AudioPlaybackEngine;句序重排;turn 冲刷;打断清缓冲;markdown/emoji strip | 端到端 <3s;句间无断音/乱序;收尾句必出声;打断后不串句;断流兜底 | 高 |
|
||||||
|
| **M4 打断+回声**(2~3d) | barge-in 状态机(管线正确性已在 M3 落地);回声对策 | 状态机;播放时 VAD 阈值上调 + 300ms 判定 + 耳机提示(v1 不做音频焦点/AEC,见 §10 YAGNI) | 播放中可打断;无严重回声误触 | 中 |
|
||||||
|
| **M5 打磨**(1~2d) | 设置页字段、超时/错误提示、文字记录化、README 限制说明、隐私告知 | 全链路错误矩阵 | 连续 10 分钟通话稳定;文字记录完整 | 低 |
|
||||||
|
|
||||||
|
依赖:M0→M1→M2→M3→M4 顺序推进;M3 的 spike 结果可能回改 M1 的格式假设;**M3 的管线正确性(§3.6)必须在 M4 打断前落地**,否则 M4 必然返工。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. 风险清单与对策
|
||||||
|
|
||||||
|
| 优先级 | 风险 | 对策 |
|
||||||
|
|---|---|---|
|
||||||
|
| P0 | 录音权限缺失 | Manifest RECORD_AUDIO + 运行时请求 + 拒绝引导 + 设备占用报错 |
|
||||||
|
| P0 | PCM 参数配错=全断音 | M3 开头真实验证(spike)后再定 AudioTrack |
|
||||||
|
| P0 | ASR/TTS 错误无兜底 | ASR 空文本→"没听清"+重录(限1次自动重试);TTS 失败→文字气泡 |
|
||||||
|
| P1 | 超时 | VAD 45s 硬切;WAITING_REPLY 20s 无首包→"网络异常"提示;TTS 断流→角标+重试 |
|
||||||
|
| P1 | 回声/假打断 | 播放时 VAD 阈值上调 + 300ms 判定 + 耳机提示;不做 AEC(Android 无公共 AEC API) |
|
||||||
|
| P1 | 前台强制机制 | "仅前台运行"需代码落地:onPause→`pauseRecording()`,防切后台录音继续、误发真人对话 |
|
||||||
|
| P1 | key 落盘明文 + 备份泄露 | SharedPreferences 是明文落盘 + `allowBackup=true` 会打进备份 → `dataExtractionRules`/`fullBackupContent` 排除 prefs;key/音频 base64 绝不进日志 |
|
||||||
|
| P1 | TTS 免费期结束 | 现"限时免费",一旦收费=无限轮 ASR+大量 TTS 计费 → 设置页显示累计通话时长/轮数;文档注明 |
|
||||||
|
| P1 | 数据/耗电 | 16k 上行 1.9MB/min、24k 下行 2.9MB/min;v1 仅前台运行(配合前台强制机制) |
|
||||||
|
| P2 | 后台/熄屏通话 | v1 明确不支持;v2 前台 Service(foregroundServiceType="microphone") |
|
||||||
|
| P2 | 音频焦点冲突 | v1 不做(来电天然打断,可接受);v2 若需再引入 AudioFocusRequest |
|
||||||
|
| P2 | 多端会话冲突 | 已存在(同 session 并发回复并进同一气泡);低频,记录即可 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. 测试策略
|
||||||
|
|
||||||
|
1. **PCM spike(M3 首件事)**:真实请求确认采样率/声道/字节序 + 鉴权头定案 → 写死前先验证
|
||||||
|
2. **单元测试**:`SentenceSplitter`(句切边界/60字硬切/长文本/clear/flush)、`CallState` 状态机(含打断时序/原子操作)、错误矩阵(ASR 空文本/HTTP 4xx/TTS 断流/未检测到语音)
|
||||||
|
3. **管线集成测试**(M3):用**含 `end` 事件的黄金帧 fixture** 驱动(验证 turn 冲刷、句序、打断清缓冲),避免只 mock repository
|
||||||
|
4. **验收量化**:debug HUD 显示 VAD/ASR/LLM/TTS 各阶段耗时;定义"≤1 次卡顿/60s、打断误触率 <5%/分钟"
|
||||||
|
5. **真机验证清单**(M4 后):端到端延迟实测、长回复流畅度、打断/回声、锁屏/来电行为、权限拒绝流程、10 分钟稳定性、数据用量、**WiFi↔蜂窝切换、飞行模式中断**
|
||||||
|
6. `sh ./gradlew :app:assembleDebug` 每次里程碑后跑(本机 JDK 覆盖与 local.properties 已配好)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. 参考实现与资料
|
||||||
|
|
||||||
|
- 参考仓库:`timyang2005/MiMoTTSReader`(纯 Kotlin,SSE 流式 + PCM→WAV 手写头,但它是收完再转文件供 Legado 朗读;需改造成逐块喂 AudioTrack)
|
||||||
|
- MiMo ASR 文档:`https://mimo.mi.com/docs/zh-CN/api/audio/Speech-Recognition`
|
||||||
|
- MiMo TTS API 参考:`https://mimo.mi.com/docs/zh-CN/api/audio/tts`
|
||||||
|
- MiMo TTS 使用指南:`https://mimo.mi.com/docs/zh-CN/quick-start/usage-guide/audio/speech-synthesis-v2.5`
|
||||||
|
- MiMo 价格:`https://mimo.mi.com/docs/zh-CN/price/pay-as-you-go`
|
||||||
|
- MiMo key:`https://platform.xiaomimimo.com/#/console/api-keys`(sk- 开头)
|
||||||
|
- 本机 AstrBot:`/home/miaomiao/Project/astrbot`(v4.27.2;`samples/stt_health_check.wav` 可作 ASR 测试样例)
|
||||||
|
- 本机 App:`/home/miaomiao/Project/rainnya-chat`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. v1 范围界定(明确不做)
|
||||||
|
|
||||||
|
- ❌ 后台/熄屏通话(v2 前台 Service)
|
||||||
|
- ❌ AEC 回声消除(无公共 API,代价大)
|
||||||
|
- ❌ 语音消息存储/重听(ChatMessage 加 audioPath 属 v2,需 DB migration)
|
||||||
|
- ❌ 多端会话并发冲突处理
|
||||||
|
- ❌ 通话记录气泡"语音"标记
|
||||||
|
- ❌ 服务端 STT/TTS 路线(未来演进方向,需 AstrBot 大改)
|
||||||
|
- ❌ 音频焦点(AudioFocusRequest)——来电天然打断,v1 可接受
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. 修订记录(红队审查合并,2026-08-23 v1.1)
|
||||||
|
|
||||||
|
红队审查(独立 oracle)检出 4 高 + 6 中 + 6 低 + 3 YAGNI,均已并入上文,要点:
|
||||||
|
|
||||||
|
- **管线正确性定稿**(新增 §3.6):打断原子操作(四件事)、end 冲刷、句序重排 + seq、wsEvents 驱动
|
||||||
|
- **M3 承载管线正确性**(打断/冲刷/句序前移),M4 收窄为"打断状态机 + 回声对策";修复"打断/管线正确性排到 M4 导致 M3 返工"的优先级误判
|
||||||
|
- 修正 TTS 鉴权头矛盾:统一 `Authorization: Bearer`(官方 + AstrBot `mimo_api_common` 一致),参考仓库 `api-key:` 仅作线索,M3 spike 定案
|
||||||
|
- §3.3 加前台强制机制(onPause→pauseRecording)+ 最小控件集;§4 VAD 加"未检测到语音"分支 + 阈值下限
|
||||||
|
- §6 新增 P1:key 落盘明文/备份泄露(排除 backup)、TTS 免费期结束(用量提示)、前台强制机制落地
|
||||||
|
- **YAGNI 砍项**:① `MiMoClient` 抽象接口删除(两具体类 + baseUrl 注入足够);② 音频焦点 v1 砍掉(来电天然打断);③ 自适应 VAD 阈值简化为固定下限 + 前 500ms 校准
|
||||||
|
- **低优先收尾项**:TTS 前 strip markdown/emoji(M3);设置页隐私告知("语音将上传 MiMo 云端");验收量化(debug HUD 分阶段耗时、≤1 卡顿/60s、打断误触率<5%/min);黄金帧 fixture 集成测试;真机清单补网络切换/飞行模式
|
||||||
|
|
||||||
|
**审查结论**:报告可据此开工;M0/M1 可立即动工,M3 先做 spike + §3.6 语义落地即消除返工风险。核心方向、里程碑划分、路线取舍成立,无需推翻。
|
||||||
Reference in New Issue
Block a user