moonfire-nvr/design/time.md

# Moonfire NVR Time Handling

Status: **current**

> A man with a watch knows what time it is. A man with two watches is never
> sure.
>
> — Segal's law

## Objective

Maximize the likelihood Moonfire NVR's timestamps are useful.

The timestamp corresponding to a video frame should roughly match timestamps
from other sources:

   * another video stream from the same camera. Given a video frame from the
     "main" stream, a video frame from the "sub" stream with a similar
     timestamp should have been recorded near the same time, and vice versa.
     This minimizes confusion when switching between views of these streams,
     and when viewing the "main" stream timestamps corresponding to a motion
     event gathered from the less CPU-intensive "sub" stream.
   * on-camera motion events from the same camera. If the video frame reflects
     the motion event, its timestamp should be roughly within the event's
     timespan.
   * streams from other cameras. Recorded views from two cameras of the same
     event should have similar timestamps.
   * events noted by the owner of the system, neighbors, police, etc., for the
     purpose of determining chronology, to the extent those persons use
     accurate clocks.

Two segments of video recorded from the same stream of the same camera should
not overlap. This would make it impossible for a user interface to present a
simple timeline for accessing all recorded video.

Durations should be useful over short timescales:

   * If an object's motion is recorded, distance travelled divided by the
     duration of the frames over which this motion occurred should reflect the
     object's average speed.
   * Motion should appear smooth. There shouldn't be excessive frame-to-frame
     jitter due to such factors as differences in encoding time or network
     transmission.

This document describes an approach to achieving these goals when the
following statements are true:

   * the NVR's system clock is within a second of correct on startup. (True
     when NTP is functioning or when the system has a real-time clock battery
     to preserve a previous correct time.)
   * the NVR's system time does not experience forward or backward "step"
     corrections (as opposed to frequency correction) during operation.
   * the NVR's system time advances at roughly the correct frequency. (NTP
     achieves this through frequency correction when operating correctly.)
   * the cameras' clock frequencies are off by no more than 500 parts per
     million (roughly 43 seconds per day).
   * the cameras are geographically close to the NVR, so in most cases network
     transmission time is under 50 ms. (Occasional delays are to be expected,
     however.)

When one or more of those statements are false, the system should degrade
gracefully: preserve what properties it can, gather video anyway, and when
possible include sufficient metadata to assess trustworthiness.

Additionally, the system should not require manual configuration of camera
frequency corrections.

## Background

Time in a distributed system is notoriously tricky. [Falsehoods programmers
believe about
time](http://infiniteundo.com/post/25326999628/falsehoods-programmers-believe-about-time)
and [More falsehoods programmers believe about time; "wisdom of the crowd"
edition](http://infiniteundo.com/post/25509354022/more-falsehoods-programmers-believe-about-time)
give a taste of the problems encountered. These problems are found even in
datacenters with expensive, well-tested hardware and relatively reliable
network connections. Moonfire NVR is meant to run on an inexpensive
single-board computer and record video from budget, closed-source cameras,
so such problems are to be expected.

Moonfire NVR typically has access to the following sources of time
information:

   * the local `CLOCK_REALTIME`. Ideally this is maintained by `ntpd`:
     synchronized on startup, and frequency-corrected during operation. A
     hardware real-time clock and battery keep accurate time across restarts
     if the network is unavailable on startup. In the worst case, the system
     has no real-time clock or no battery and a network connection is
     unavailable. The time is far in the past on startup and is never
     corrected or is corrected via a step while Moonfire NVR is running.
   * the local `CLOCK_MONOTONIC`. This should be frequency-corrected by `ntpd`
     and guaranteed to never experience "steps", though its reference point is
     unspecified.
   * the local `ntpd`, which can be used to determine if the system is
     synchronized to NTP and quantify the precision of synchronization.
   * each camera's clock. The ONVIF specification mandates cameras must
     support synchronizing clocks via NTP, but in practice cameras appear to
     use SNTP clients which simply step time periodically and provide no
     interface to determine if the clock is currently synchronized. This
     document's author owns several cameras with clocks that run roughly 20
     ppm fast (2 seconds per day) and are adjusted via steps.
   * the RTP timestamps from each of a camera's streams. As described in [RFC
     3550 section 5.1](https://tools.ietf.org/html/rfc3550#section-5.1), these
     are monotonically increasing with an unspecified reference point. They
     can't be directly compared to other cameras or other streams from the
     same camera. Emperically, budget cameras don't appear to do any frequency
     correction on these timestamps.
   * in some cases, RTCP sender reports, as described in [RFC 3550 section
     6.4](https://tools.ietf.org/html/rfc3550#section-6.4). These correlate
     RTP timestamps with the camera's real time clock. However, these are only
     sent periodically, not necessarily at the beginning of the session.
     Some cameras omit them entirely depending on firmware version, as noted
     in [this forum post](http://www.cctvforum.com/viewtopic.php). Additionally,
     Moonfire NVR currently uses ffmpeg's libavformat for RTSP protocol
     handling; this library exposes these reports in a limited fashion.

The camera records video frames as in the diagram below:

![Video frame timeline](time-frames.png)

Each frame has an associated RTP timestamp. It's unclear from skimming RFC
3550 exactly what time this represents, but it must be some time after the
last frame and before the next frame. At a typical rate of 30 frames per
second, this timespan is short enough that this uncertainty won't be the
largest source of time error in the system. We'll assume arbitrarily that the
timestamp refers to the start of exposure.

RTP doesn't transmit the duration of each video frame; it must be calculated
from the timestamp of the following frame. This means that if a stream is
terminated, the final frame has unknown duration.

As described in [schema.md](schema.md), Moonfire NVR saves RTSP video streams
into roughly one-minute "recordings", with a fixed rotation offset after the
minute in the NVR's wall time.

## Overview

Moonfire NVR will use the RTP timestamps to calculate video frames' durations.
For the first segment of video, it will trust these completely. It will use
them and the NVR's wall clock time to establish the start time of the
recording. For following segments, it will slightly adjust durations to
compensate for difference between the frequencies of the camera and NVR
clock, trusting the latter to be accurate.

## Detailed design

On every frame of video, Moonfire NVR will get a timestamp from
`CLOCK_MONOTONIC`. On the first frame, it will additionally get a timestamp
from `CLOCK_REALTIME` and compute the difference. It uses these to compute a
monotonically increasing real time of receipt for every frame, called the
_local frame time_. Assuming the local clock is accurate, this time is an
upper bound on when the frame was generated. The difference is the sum of the
following items:

   * H.264 encoding
   * buffering on the camera (particularly when starting the stream—some
     cameras apparently send frames that were captured before the RTSP session
     was established)
   * network transmission time

These values may produce some jitter, so the local frame time is not directly
used to calculate frame durations. Instead, they are primarily taken from
differences in RTP timestamps from one frame to the next. During the first
segment of video, these RTP timestamp differences are used directly, without
correcting for incorrect camera frequency. At the design limit of 500 ppm
camera frequency error, and an upper bound of two minutes of recording for the
initial segment (explained below), this causes a maximum of 60 milliseconds of
error.

The _local start time_ of a segment is calculated when ending it. It's defined
as the minimum for all frames of the local frame time minus the duration of
all previous frames. If there are many frames, this means neither initial
buffering nor spikes of delay in H.264 encoding or network transmission cause
the local start time to become inaccurate. The least delayed frame wins.

The first segment either ends with the RTSP session (due to error/shutdown) or
on rotation. In the former case, there may not be many samples to use in
calculating the local start time; accuracy may suffer but the system degrades
gracefully. Rotation doesn't happen until the second time the rotation offset
is passed, so rotation happens after 1–2 minutes rather than 0–1 minutes to
maximize accuracy.

The _start time_ of the first segment is its local start time. The start time
of following segments is the end time of the previous segment.

The duration of following segments is adjusted to compensate for camera
frequency error, assuming the NVR clock's frequency is more trustworthy. This
is done as follows. The _local duration_ of segment _i_ is calculated as the
local start time of segment _i+1_ minus the local start time of segment _i_.
The _cumulative error_ as of segment _i_ is defined as the local duration of
all previous segments minus the duration of all previous segments. The
duration of segment _i_ should be adjusted by up to 500 ppm to eliminate
cumulative error. (For a one-minute segment, this is 0.3 ms, or 27 90kHz units.)
This correction should be spread out across the segment to minimize jitter.

Each segment's local start time is also stored in the database as a delta to
the segment's start time. These stored values aren't  for normal system
operation but may be handy in understanding and correcting errors.

## Caveats

### Stream mismatches

There's no particular reason to believe this will produce perfectly matched
streams between cameras or even of main and sub streams within a camera.
If this is insufficient, there's an alternate calculation of start time that
could be used in some circumstances: the _camera start time_. The first RTCP
sender report could be used to correlate a RTP timestamp with the camera's
wall clock, and thus calculate the camera's time as of the first frame.

The _start time_ of the first segment could be either its local start time or
its camera start time, determined via the following rules:

   1. if there is no camera start time (due to the lack of a RTCP sender
      report), the local start time wins by default.
   2. if the camera start time is before 2016-01-01 00:00:00 UTC, the local
      start time wins.
   3. if the local start time is before 2016-01-01 00:00:00 UTC, the camera
      start time wins.
   4. if the times differ by more than 5 seconds, the local start time wins.
   5. otherwise, the camera start time wins.

These rules are a compromise. When a system starts up without NTP or a clock
battery, it typically reverts to a time in the distant past. Therefore times
before Moonfire NVR was written should be checked for and avoided. When both
systems have a believably recent timestamp, the local time is typically more
accurate, but the camera time allows a closer match between two streams of
the same camera.

This still doesn't completely solve the problem, and it's unclear it is even
better. When using camera start times, different cameras' streams may be
mismatched by up twice the 5-second threshold described above. This could even
happen for two streams within the same camera if a significant step happens
between their establishment. More frequent SNTP adjustments may help, so that
individual steps are less frequent. Or Moonfire NVR could attempt to address
this with more complexity: use sender reports of established RTSP sessions to
detect and compensate for these clock splits.

It's unclear if these additional mechanisms are desirable or worthwhile. The
simplest approach will be adopted initially and adapted as necessary.

### Time discontinuities

If the local system's wall clock time jumps during a recording ([as has
happened](https://github.com/scottlamb/moonfire-nvr/issues/9#issuecomment-322663674)),
Moonfire NVR will continue to use the initial wall clock time for as long as
the recording lasts. This can result in some unfortunate behaviors:

   * a recording that lasts for months might have an incorrect time all the
     way through because `ntpd` took a few minutes on startup.
   * two recordings that were in fact simultaneous might be recorded with very
     different times because a time jump happened between their starts.

It might be better to use the new time (assuming that ntpd has made a
correction) retroactively. This is unimplemented, but the
`recording_integrity` database table has a `wall_time_delta_90k` field which
could be used for this purpose, either automatically or interactively.

It would also be possible to split a recording in two if a "significant" time
jump is noted, or to allow manually restarting a recording without restarting
the entire program.

### Leap seconds

UTC time is defined as the seconds since epoch _excluding
leap seconds_. Thus, timestamps during the leap second are ambiguous, and
durations across the leap second should be adjusted.

In POSIX, the system clock (as returned by `clock_gettime(CLOCK_REALTIME,
...`) is defined as representing UTC. Note that some
systems may instead be following a [leap
smear](https://developers.google.com/time/smear) policy in which instead of
one second happening twice, the clock runs slower. For a 24-hour period, the
clock runs slower by a factor of 1/86,400 (an extra ~11.6 μs/s).

In Moonfire NVR, all wall times in the database are based on UTC as reported
by the system, and it's assumed that `start + duration = end`. Thus, a leap
second is similar to a one-second time jump (see "Time discontinuities"
above).

Here are some options for improvement:

#### Use `clock_gettime(CLOCK_TAI, ...)` timestamps

Timestamps in the TAI clock system don't skip leap seconds. There's a system
interface intended to provide timestamps in this clock system, and Moonfire
NVR could use it. Unfortunately this has several problems:

   * `CLOCK_TAI` is only available on Linux. It'd be preferable to handle
     timestamps in a consistent way on other platforms. (At least on macOS,
     Moonfire NVR's current primary development platform.)
   * `CLOCK_TAI` is wrong on startup and possibly adjusted later. The offset
     between TAI and UTC is initially assumed to be 0. It's corrected when/if
     a sufficiently new `ntpd` starts.
   * We'd need a leap second table to translate this into calendar time. One
     would have to be downloaded from the Internet periodically, and we'd need
     to consider the case in which the available table is expired.
   * `CLOCK_TAI` likely doesn't work properly with leap smear systems. Where
     the leap smear prevents a time jump for `CLOCK_REALTIME`, it likely
     introduces one for `CLOCK_TAI`.

#### Use a leap second table when calculating differences

Moonfire NVR could retrieve UTC timestamps from the system then translate then
to TAI via a leap second table, either before writing them to the database or
whenever doing math on timestamps.

As with `CLOCK_TAI`, this would require downloading a leap second table from
the Internet periodically.

This would mostly solve the problem at the cost of complexity. Timestamps
obtained from the system for a two-second period starting with each leap
second would still be ambiguous.

#### Use smeared time

Moonfire NVR could make no code changes and ask the system administrator to
use smeared time. This is the simplest option. On a leap smear system, there
are no time jumps. The ~11.6 ppm frequency error and the maximum introduced
absolute error of 0.5 sec can be considered acceptable.

Alternatively, Moonfire NVR could assume a specific leap smear policy (such as
24-hour linear smear from 12:00 the day before to 12:00 the day after) and
attempt to correct the time into TAI with a leap second table. This behavior
would work well on a system with the expected configuration and produce
surprising results on other systems. It's unfortunate that there's no standard
way to determine if a system is using a leap smear and with what policy.