# Enhancements: Speech (Live Captions & Translations) OptiView Live can automatically transcribe the spoken audio of your live stream and translate it into one or more languages, delivered to your viewers as text tracks (captions) in real time and in sync with the video. This is done through the **`speech` enhancement**, configured on an ingest. > **Premium Feature** > > This feature is a premium option that must be enabled on your account. If you would like to use this capability on your project, please [submit a support ticket](https://support.dolby.io/hc/en-au) or reach out to your [sales/solutions](https://optiview.dolby.com/contact/) contact. As long as the feature is not enabled on your account, requests that configure a speech enhancement fail with the error code `translations_not_enabled`. ## How it works You enable the speech enhancement on an **ingest** by adding an entry with `"kind": "speech"` to its `enhancements` list. You tell us which language is spoken in the stream (`sourceLanguage`) and which languages you want translated captions in (`targetTextTracks`). The audio is transcribed, translated into the requested target languages and injected back into the stream as text tracks, aligned with what is being said. Optionally, a **dictionary** with domain-specific terms can be attached to improve the speech recognition. ### What your viewers get * A caption track in the **source language** is always included, followed by one track per language in `targetTextTracks`, in the order you listed them. * On HLS, each language is exposed as a `SUBTITLES` rendition in the multivariant playlist (IMSC1 TTML, `stpp`). The source-language track comes first and is marked as the default. * The same tracks remain available in the DVR window and in the hourly VOD recordings of the channel. ### Dashboard You don't need the API to set this up: the ingest form in the dashboard has a **Speech** toggle where you can pick the source language, target text tracks, AI context, latency mode, caption position and dictionary. ## Quick start 1. Make sure the ingest's `region` supports speech enhancements (see [Regions](#regions)). 2. Look up the supported languages: * [`GET /v2/enhancements/speech/supported-source-languages`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/get-supported-source-languages.md) * [`GET /v2/enhancements/speech/supported-target-languages`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/get-supported-target-languages.md) 3. (Optional) Create a dictionary with domain-specific terms: [`POST /v2/enhancements/speech/dictionaries`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/create-speech-dictionary.md). 4. Add a `speech` entry to the `enhancements` list when creating or updating an ingest. ## Enabling the speech enhancement on an ingest The speech enhancement is configured per ingest, either when creating the ingest or by updating an existing one: * [`POST /v2/channels/{channelId}/ingests`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/create-channel-ingest.md) — create an ingest with a speech enhancement * [`PATCH /v2/ingests/{ingestId}`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/update-ingest.md) — add, change or remove the speech enhancement on an existing ingest ```json { "name": "my-ingest", "type": "rtmp-push", "region": "oke-production-lon-1", "enhancements": [ { "kind": "speech", "sourceLanguage": "en", "targetTextTracks": ["fr", "de", "es"], "latencyMode": "fast", "extraPrompt": "this is a live horse race", "position": { "bottom": 10 }, "dictionary": "my-dictionary-id" } ] } ``` Use a `region` id as returned by [`GET /v2/regions`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/get-regions.md). ### `speech` entry fields The full schema is documented in the API reference (`SpeechEnhancement` on [create ingest](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/create-channel-ingest.md) and [update ingest](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/update-ingest.md)). In short: | Field | Type | Required | Description | | ------------------ | ---------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sourceLanguage` | `string` | yes | Language spoken in the stream (must be a supported source language) | | `targetTextTracks` | `string[]` | yes | Languages to translate into (1 to **5**, must be supported target languages). A caption track in the source language is always included, so don't list the source language or a variant of it (e.g. `en-US` for `en`) | | `sourceAudioTrack` | `number` | no | PID of the audio track to use as translation source (non-negative integer) | | `dictionary` | `string` | no | ID of a dictionary that helps the speech recognition. The dictionary's `language` must match the enhancement's `sourceLanguage`. Set to `null` to disconnect the dictionary | | `latencyMode` | `string` | no | Trade-off between caption delay and translation quality: `fastest`, `fast` (default), `balanced` or `highest`. `highest` waits longest for the best result | | `extraPrompt` | `string` | no | Extra context for the AI, applied to every translation (at most 1000 characters), e.g. `"this is a live horse race"` or `"remove profane language from the output"`. Set to `null` to remove it | | `position` | `object` | no | Position of the rendered captions. `position.bottom` is an integer from 0 to 90: the margin from the bottom of the viewport, in percent of the viewport height | ### Updating and disabling * On update, the `enhancements` list is replaced as a whole: always send the full list you want to keep. * To disable the speech enhancement, update the ingest with an `enhancements` list without a `speech` entry (e.g. `[]`). * To disconnect a dictionary, resend the `speech` entry without the `dictionary` field (or with `"dictionary": null`). * `captions` and `enhancements` can't be changed while an engine of the ingest is running: the update fails with `409 engine_playing`. Stop the channel (or its engines) first. ### Connection details With a speech enhancement, the stream is first routed through the captioning service. This affects how you connect your encoder: * **`rtmp-push`** and **`srt-push`**: always push to the ingest URL returned by the API for that ingest — it differs from a regular ingest's connection details (an `srt-push` ingest with a speech enhancement, for example, returns a URL without a stream key). * **`srt-pull`**: the captioning service pulls the ingest's `url` itself, so your source must be reachable from the captioning service. ## Regions A speech enhancement requires a `region`, and the region must be one where speech enhancements are available: * A missing `region` fails with `invalid_value`. * A region that doesn't support speech enhancements fails with `speech_enhancement_not_supported`. Use [`GET /v2/regions`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/get-regions.md) to list the regions available to your organization. ## Supported languages Use these endpoints to discover which languages you can translate from and to. Only languages returned by these endpoints are accepted in `sourceLanguage` and `targetTextTracks`: | Method | Path | Description | | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | | `GET` | [`/v2/enhancements/speech/supported-source-languages`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/get-supported-source-languages.md) | Languages the speech recognition can transcribe (`sourceLanguage`) | | `GET` | [`/v2/enhancements/speech/supported-target-languages`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/get-supported-target-languages.md) | Languages that can be translated into (`targetTextTracks`) | Both return a simple list: ```json { "data": ["en", "fr", "de", "es", "nl"] } ``` ## Dictionaries A speech dictionary is a **flat list of terms** — names, proper nouns, acronyms and specialized terminology — that helps the speech recognition correctly caption the source audio of a speech enhancement. It does **not** translate terms: the source audio is first captioned in the source language (aided by the dictionary) and then translated into the target languages. Connect a dictionary to an ingest via the `dictionary` field of the speech enhancement entry; the dictionary's `language` must match that enhancement's `sourceLanguage`. | Method | Path | Description | | -------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | `GET` | [`/v2/enhancements/speech/dictionaries`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/get-speech-dictionaries.md) | List dictionaries (supports `cursor`, `limit` and `name` query parameters) | | `POST` | [`/v2/enhancements/speech/dictionaries`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/create-speech-dictionary.md) | Create a dictionary | | `GET` | [`/v2/enhancements/speech/dictionaries/{id}`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/get-speech-dictionary.md) | Get a single dictionary | | `PATCH` | [`/v2/enhancements/speech/dictionaries/{id}`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/update-speech-dictionary.md) | Update a dictionary | | `DELETE` | [`/v2/enhancements/speech/dictionaries/{id}`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/delete-speech-dictionary.md) | Delete a dictionary | Example create request: ```json { "name": "horse-racing", "language": "en", "terms": ["OptiView", "A. Väyrynen", "D&I|dee-and-eye,dee-n-eye"] } ``` ### Writing good dictionary terms * Keep terms short: each entry is a single term of at most **6 words**. One- to three-word phrases give the best results. * Add terms the way they are actually spoken. An entry like `"Initial Public Offering (IPO)"` won't match anything — a speaker says either the full phrase or just "IPO", so add those as two separate terms. * When a term is pronounced differently than its spelling suggests, append one or more **phonetic "sounds like" hints** after a pipe, separated by commas: `"D&I|dee-and-eye,dee-n-eye"`. * Terms containing symbols, digits or punctuation (hashtags, abbreviations, hyphenated words) should always get a phonetic hint, since their spelling alone gives the speech recognition little to work with. * Dictionary terms are hints, not overrides: when the speech recognition is confident it heard a different word, it captions that instead. ### Dictionary rules * `language` must be a valid IETF BCP 47 language tag (e.g. `"en"`, `"en-US"`) and must match the `sourceLanguage` of the speech enhancement it is connected to. * Terms are cleaned up automatically where possible: whitespace is normalized and empty and duplicate entries are dropped. Terms that cannot be fixed automatically are rejected with `invalid_value`, describing the offending entry: terms containing the forbidden characters `;` or `/`, terms with more than 6 words, more than one `|` separator, or a missing term before the `|`. * On update, the `terms` list is replaced as a whole. * A dictionary cannot be deleted while it is connected to an ingest that is in use by a running engine; the delete fails with `409 speech_dictionary_in_use`. Once deleted, the dictionary is automatically disconnected from all ingests that referenced it. ## Usage Usage of the speech enhancement is reported in **minutes**: captioned minutes (transcription of the source audio) and translated minutes (one minute per target language per minute of audio). Use these to reconcile what you are billed: | Scope | Captioned minutes | Translated minutes | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Organization | [`GET /v2/enhancements/speech/analytics/captioned-minutes`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/get-speech-captioned-minutes.md) | [`GET /v2/enhancements/speech/analytics/translated-minutes`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/get-speech-translated-minutes.md) | | Channel | [`GET /v2/channels/{id}/analytics/captions/captioned-minutes`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/get-channel-captions-captioned-minutes.md) | [`GET /v2/channels/{id}/analytics/captions/translated-minutes`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/get-channel-captions-translated-minutes.md) | | Ingest | [`GET /v2/ingests/{id}/analytics/captions/captioned-minutes`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/get-ingest-captions-captioned-minutes.md) | [`GET /v2/ingests/{id}/analytics/captions/translated-minutes`](https://docs-preview.optiview.dolby.com/pr-912/theolive/api/get-ingest-captions-translated-minutes.md) | All of these take `start` and `end` dates (`YYYY-MM-DD`, interpreted as UTC) and an optional `resolution` (`day` or `month`). ## Limitations * Translated captions are currently only available on **HLS** streams. They are **not** available on HLS-TS or HESP. * The ingest's `region` must support speech enhancements (see [Regions](#regions)). * `targetTextTracks` can contain at most **5** languages and must not include the source language (or a variant of it). * The `enhancements` list can contain at most **1** `speech` entry. * The speech enhancement is only supported on ingests of type `rtmp-push`, `srt-push` and `srt-pull`. * `sourceLanguage` and every language in `targetTextTracks` must be one of the supported languages (see above). * `captions` and `enhancements` cannot be changed while an engine of the ingest is running. ## Errors | Code | Meaning | | --------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `translations_not_enabled` | Live captions & translations is not enabled on your account — contact your account team | | `speech_enhancement_not_supported` | The ingest's `region` does not support speech enhancements | | `invalid_value` | Invalid configuration (wrong ingest type, missing `region`, missing/unsupported languages, source language in `targetTextTracks`, invalid dictionary terms) | | `too_many_values` | More than 5 target text tracks were provided | | `engine_playing` (409) | `captions` or `enhancements` were changed while an engine of the ingest is running | | `speech_dictionary_in_use` (409) | The dictionary is connected to an ingest with a running engine and cannot be deleted | | `speech_dictionary_create_failed` / `speech_dictionary_update_failed` / `speech_dictionary_delete_failed` | The dictionary could not be synchronized with the captioning provider — retry, or contact support if the problem persists |