Maintenance windows

Many channels keep their engines running for weeks or months. The servers underneath them occasionally have to be replaced: security patches, hardware refreshes, moving to a newer platform. Replacing a server means restarting every engine on it, and that interrupts the stream for a short moment.

Maintenance windows let you decide when such an interruption is acceptable. You configure recurring time slots for your organization, and we only ever perform maintenance inside those slots. Outside them, nothing is touched.

What maintenance means

A maintenance is not an upgrade and not scheduled downtime. It moves an engine or an ingest to fresh infrastructure, with exactly the same configuration, and only when that is actually needed. Most windows pass without anything happening.

There are two kinds:

	Engine maintenance	Ingest maintenance
What is replaced	The server under a long-running engine	The server of the shared ingest infrastructure that receives a push or pull ingest (rtmp-push, srt-push, rtmp-pull, srt-pull)
What happens in the window	The engine is restarted with the same configuration and software version	The ingest's connection is moved to another ingest server
What your viewers see	The stream ends cleanly (players reach the end of the stream, not an error) and resumes once the encoder has reconnected	A short gap of a few seconds; the stream does not end
What your encoder sees	It loses its connection and reconnects to the same ingest URL	It is disconnected once and reconnects to the same ingest URL. For a pull ingest, we fetch the source again ourselves
Typical duration	One to two minutes until the stream is back	A few seconds, as soon as the encoder reconnects
Reported on	The engine's maintenance field (or the ingest's, for rtsp-pull ingests, which are started and stopped as a whole)	The ingest's maintenance field
Webhook events	engine.maintenance.*	ingest.maintenance.*
reason	node-rotation	ingest-server-retirement

Playback URLs, ingest URLs, the engine's configuration and its software version never change. The engine run that was open when the maintenance started stays open: the interruption is part of the same run, not a new one, and it is not billed differently.

Rules we always follow

• Only inside a window, and never in the last five minutes of one, so a maintenance always has time to finish before the window closes.
• Only when needed. A window is an opportunity, not an appointment.
• One engine per channel at a time. A channel with a backup engine keeps one engine up throughout; its engines are restarted one after the other, never together. An engine restart and an ingest move never happen on the same channel at the same time either.
• Only running engines. A stopped engine has nothing to maintain; starting an engine always lands it on fresh infrastructure.

Configuring windows

A maintenance window is a recurring time slot in UTC:

Setting	Meaning
dayOfWeek	The day the window starts: monday … sunday.
startTime	Start of the window, HH:mm in 24-hour format (e.g. 02:00).
durationMinutes	Length of the window, from 15 minutes up to 24 hours (1440). A window may run past midnight into the next day.
weeksOfMonth	Optional. Restrict the window to the n-th occurrence of that day in the month: any of 1, 2, 3, 4 and last. [2] means "the second Tuesday of the month", [1, 3] roughly every other week. Omitted means every week.

You can combine up to seven windows. Together they must:

• offer a maintenance opportunity at least once every 35 days, so that any "once a month" schedule is accepted, but nothing sparser;
• not overlap each other.

Changes take effect immediately: if you have an event coming up during one of your windows, change the windows beforehand.

Some examples:

You want	Configure
Every Tuesday from 02:00 to 03:00 UTC	{ "dayOfWeek": "tuesday", "startTime": "02:00", "durationMinutes": 60 }
The first and third Sunday of the month from 04:00 to 06:00	{ "dayOfWeek": "sunday", "startTime": "04:00", "durationMinutes": 120, "weeksOfMonth": [1, 3] }
Every night from 03:00 to 03:15 UTC	Seven windows, one per dayOfWeek, each with "startTime": "03:00" and "durationMinutes": 15
Saturday night to Sunday morning, 23:00 to 05:00, last week only	{ "dayOfWeek": "saturday", "startTime": "23:00", "durationMinutes": 360, "weeksOfMonth": ["last"] }

In the dashboard

1. Navigate to Streaming → Maintenance windows.
2. The Next maintenance opportunity card shows the next (or currently ongoing) window and the last maintenance we performed.
3. Under Windows, click Add window and fill in the row:
   • Recurrence — Every week, or Specific weeks to pick the Weeks of the month (1st, 2nd, 3rd, 4th, Last).
   • Day (UTC) and Start (UTC) — When the window starts.
   • Duration (minutes) — From 15 minutes to 24 hours, in steps of 15.
4. Repeat for up to seven windows, remove a window with its trash icon, then click Save.

Validation errors, such as overlapping windows or a gap of more than 35 days between opportunities, are shown on the form.

When something needs maintenance, a Maintenance section at the top of the page lists the engines and ingests that are waiting for the next window or being maintained right now, each linking to its channel.

API

Maintenance windows are managed per organization through GET and PUT /v2/maintenance-windows (see the REST API). PUT replaces the whole list.

PUT https://api.theo.live/v2/maintenance-windows

{
  "windows": [
    { "dayOfWeek": "tuesday", "startTime": "02:00", "durationMinutes": 60 },
    { "dayOfWeek": "sunday", "startTime": "04:00", "durationMinutes": 120, "weeksOfMonth": [1, 3] }
  ]
}

Both GET and PUT return the organization's maintenance settings, together with what is pending or in progress:

{
  "data": {
    "windows": [
      { "dayOfWeek": "tuesday", "startTime": "02:00", "durationMinutes": 60 },
      { "dayOfWeek": "sunday", "startTime": "04:00", "durationMinutes": 120, "weeksOfMonth": [1, 3] }
    ],
    "nextWindow": { "start": "2026-10-27T02:00:00.000Z", "end": "2026-10-27T03:00:00.000Z" },
    "lastMaintenanceAt": "2026-09-15T02:04:12.000Z",
    "pending": { "count": 2, "engineIds": ["eng_abc123"], "ingestIds": ["ing_def456"] },
    "inProgress": { "count": 0, "engineIds": [], "ingestIds": [] }
  }
}

Field	Description
windows	The configured windows.
nextWindow	The next, or currently ongoing, occurrence of your windows.
lastMaintenanceAt	When we last performed a maintenance for your organization. Absent when we never did.
pending	Maintenance that is needed and will be performed in the next window: the number of engines and ingests concerned and their ids.
inProgress	Maintenance that is being performed right now.

In pending and inProgress, ingestIds lists both the push and pull ingests that have to move to another ingest server, and the rtsp-pull ingests whose engine has to be restarted. count is the total of both lists.

Following a maintenance

Statuses

An engine or ingest that needs, undergoes or recently finished a maintenance carries a maintenance object. A finished maintenance stays visible for 24 hours; an engine or ingest without one has no maintenance field.

"maintenance": {
  "id": "mnt_7d5a…",
  "status": "pending",
  "reason": "node-rotation",
  "markedAt": "2026-10-18T14:00:00.000Z",
  "window": { "start": "2026-10-20T02:00:00.000Z", "end": "2026-10-20T03:00:00.000Z" }
}

Field	Description
id	Identifier of the maintenance. The webhook events carry it as maintenanceId.
status	See below.
reason	Why maintenance is needed: node-rotation for engines, ingest-server-retirement for ingests.
markedAt	When the need for maintenance was detected.
window	While pending, the window in which the maintenance is planned; afterwards, the window in which it was performed.
startedAt	When the maintenance started.
completedAt	When the maintenance finished, whatever the outcome.

Status	Dashboard badge	Meaning
pending	Maintenance pending	Maintenance is needed and will be performed in the next window. Nothing has changed yet.
in_progress	Maintenance in progress	The engine is being restarted, or the ingest moved. The stream is briefly interrupted and the encoder reconnects.
completed	Maintenance completed	The engine is back and producing output again (waiting, ingesting or playing), or the ingest is live on another server.
failed	Maintenance failed	The engine did not come back by itself within about 15 minutes, or the ingest's encoder did not reconnect within 15 minutes. Our team is alerted. Starting the engine again, or restarting the encoder, resolves it on your side too.
cancelled	(none)	The maintenance is no longer needed: the engine was stopped in the meantime, the pending ingest reconnected to another server by itself, or the ingest was deleted. Nothing happened and nothing will.

In the dashboard, the badge sits next to the status of the engine or ingest on the channel details page, and in the Maintenance section of the Maintenance windows page. Hover it for the reason, when the need was detected, the planned window, and the start and end times.

One maintenance, step by step

1. Days before: the engine or ingest is flagged. Its maintenance.status is pending, with the planned window, and you receive engine.maintenance.required (or ingest.maintenance.required). Nothing changes for your viewers.
2. In the window: the maintenance starts (in_progress, engine.maintenance.started).
   • Engine: the stream ends cleanly for viewers and your encoder loses its connection. Within roughly one to two minutes the engine is back with the same settings and the same ingest URL; the encoder reconnects and the stream resumes. The engine reports provisioning, starting, waiting and then ingesting and playing again, with the usual engine webhook events.
   • Ingest: the encoder is disconnected and reconnects to the same URL on another ingest server. The engine behind it keeps running and sees a short gap in its input.
3. Done: completed, with engine.maintenance.completed.

:::tip Taking matters into your own hands A fresh start always lands on fresh infrastructure. If you would rather not wait for your window, stop and start a flagged engine yourself at a moment of your choosing: the pending maintenance is canceled. Likewise, reconnecting the encoder of a flagged push ingest moves it to a fresh ingest server and clears the flag. :::

Webhooks

Every step sends a webhook event: engine.maintenance.required, .started, .completed, .failed and .cancelled for engines, and the ingest.maintenance.* equivalents for ingests. The object is the engine or ingest, its parent the channel. data carries the same fields as the maintenance object, under slightly different names:

{
  "type": "engine.maintenance.started",
  "createdAt": 1761530520,
  "object": {
    "type": "engine",
    "id": "eng_abc123",
    "name": "primary",
    "organizationId": "org_xyz",
    "parent": { "type": "channel", "id": "ch_abc123", "name": "my-channel" }
  },
  "data": {
    "maintenanceId": "mnt_7d5a…",
    "status": "in_progress",
    "reason": "node-rotation",
    "markedAt": "2026-10-18T14:00:00.000Z",
    "window": { "start": "2026-10-27T02:00:00.000Z", "end": "2026-10-27T03:00:00.000Z" },
    "startedAt": "2026-10-27T02:02:00.000Z",
    "engineRunId": "run_123"
  }
}

engineRunId (engine events only) is the run the maintenance happened in; engine.maintenance.failed and ingest.maintenance.failed add an error. The events also appear in the dashboard's Logs explorer, and can be selected in the webhook form under the engine and ingest event groups.