Merge branch 'master' into new-schema

design/api.md

@@ -142,6 +142,7 @@ The `application/json` response will have a JSON object as follows:
 *   `signals`: a list of all *signals* known to the server. Each is a JSON
     object with the following properties:
     *   `id`: an integer identifier.
+    *   `source`: a UUID representing the signal source (could be a camera UUID)
     *   `shortName`: a unique, human-readable description of the signal
     *   `cameras`: a map of associated cameras' UUIDs to the type of association:
         `direct` or `indirect`. See `db/schema.sql` for more description.
@@ -340,6 +341,13 @@ arbitrary order. Each recording object has the following properties:
 *   `videoSamples`: the number of samples (aka frames) of video in this
     recording.
 *   `sampleFileBytes`: the number of bytes of video in this recording.
+*   `hasTrailingZero`: the final frame of the final recording id described by
+    this row (`endId` if present, otherwise `startId`) has a duration of 0.
+    A frame's duration is calculated by subtracting its timestamp from the
+    following frame's timestamp. When a run ends, there's no following frame
+    and Moonfire NVR fills in a duration of 0. When using `/view.mp4`, it's
+    not possible to append additional segments after such frames, as noted
+    below.
 
 Under the property `videoSampleEntries`, an object mapping ids to objects with
 the following properties:
@@ -438,7 +446,7 @@ Example request URI to retrieve recording id 1, skipping its first 26
 90,000ths of a second:
 
 ```
-    /api/cameras/fd20f7a2-9d69-4cb3-94ed-d51a20c3edfe/main/view.mp4?s=1.26
+    /api/cameras/fd20f7a2-9d69-4cb3-94ed-d51a20c3edfe/main/view.mp4?s=1.26-
 ```
 
 Note carefully the distinction between *wall duration* and *media duration*.
@@ -446,8 +454,27 @@ It's normal for `/view.mp4` to return a media presentation with a length
 slightly different from the *wall duration* of the backing recording or
 portion that was requested.
 
-TODO: error behavior on missing segment. It should be a 404, likely with an
-`application/json` body describing what portion if any (still) exists.
+Bugs and limitations:
+
+*   If the `s=` parameter references a recording id that doesn't exist when the
+    server starts processing the `/view.mp4` request, the server will return a
+    `404` with a text error message. This commonly happens when the oldest
+    recording was deleted between the `/recordings` request and the `/view.mp4`
+    request. The server probably should return a structured JSON document
+    describing exactly which recordings have been deleted. For now, the client
+    will have to retry from `/recordings` and again race against deletion.
+*   If a recording is deleted after the server starts processing `/view.mp4`
+    but before the request advances to the recording's byte position, the server
+    will abruptly drop the HTTP connection. The client must then retry to see
+    a proper 404 error. It'd be better if the server would prevent recordings
+    from being deleted while there are `/view.mp4` requests in progress which
+    reference them.
+*   The final recording in every "run" ends with a frame that has duration 0.
+    It's not possible to append additional segments after such a frame;
+    the server will return a 400 error like `Invalid argument: unable to append
+    recording 2/16672 after recording 2/16671 with trailing zero`. See also
+    `hasTrailingZero` above, and
+    [#178](https://github.com/scottlamb/moonfire-nvr/issues/178).
 
 ### `GET /api/cameras/<uuid>/<stream>/view.mp4.txt`
 
@@ -485,11 +512,11 @@ Expected query parameters:
 *   `s` (one or more): as with the `.mp4` URL.
 
 It's recommended that each `.m4s` retrieval be for at most one Moonfire NVR
-recording segment. The fundamental reason is that the Media Source Extension
-API appears structured for adding a complete segment at a time. Large media
-segments thus impose significant latency on seeking. Additionally, because of
-this fundamental reason Moonfire NVR makes no effort to make multiple-segment
-`.m4s` requests practical:
+recording. The fundamental reason is that the Media Source Extension API appears
+structured for adding a complete segment at a time. Large media segments thus
+impose significant latency on seeking. Additionally, because of this fundamental
+reason Moonfire NVR makes no effort to make multiple-segment `.m4s` requests
+practical:
 
 *   There is currently a hard limit of 4 GiB of data because the `.m4s` uses a
     single `moof` followed by a single `mdat`; the former references the
@@ -520,6 +547,7 @@ to one or more frames of video. The first message is guaranteed to start with a
 "key" (IDR) frame; others may not. The message will contain HTTP headers
 followed by by a `.mp4` media segment. The following headers will be included:
 
+*   `X-Video-Sample-Entry-Id`: An id to use when fetching an initialization segment.
 *   `X-Recording-Id`: the open id, a period, and the recording id of the
     recording these frames belong to.
 *   `X-Recording-Start`: the timestamp (in Moonfire NVR's usual 90,000ths
@@ -603,7 +631,8 @@ streams simultaneously as well as making other simultaneous HTTP requests.
 
 Returns a `.mp4` suitable for use as a [HTML5 Media Source Extensions
 initialization segment][init-segment]. The MIME type will be `video/mp4`, with
-a `codecs` parameter as specified in [RFC 6381][rfc-6381].
+a `codecs` parameter as specified in [RFC 6381][rfc-6381]. The `<id>` should be a value
+previously extracted from the `X-Video-Sample-Entry-Id` header returned in a `.../live.m4s` response.
 
 An `X-Aspect` HTTP header will include the aspect ratio as width:height,
 eg `16:9` (most cameras) or `9:16` (rotated 90 degrees).

design/glossary.md

@@ -1,5 +1,14 @@
 # Moonfire NVR Glossary
 
+*GOP:* Group of Pictures, as
+[described](https://en.wikipedia.org/wiki/Group_of_pictures) on wikipedia.
+Each GOP starts with an "IDR" or "key" frame which can be decoded by itself.
+Commonly all other frames in the GOP are encoded in terms of the frames before,
+so decoding frame 5 requires decoding frame 1, 2, 3, and 4. Many security
+cameras produce a new IDR frame (thus start a new GOP) at a fixed interval of
+1 or 2 seconds. Some cameras that use "smart encoding" or "H.264+" may produce
+GOPs that vary in length, up to several seconds.
+
 *media duration:* the total duration of the actual samples in a recording. These
 durations are based on the camera's clock. Camera clocks can be quite
 inaccurate, so this may not match the *wall duration*. See [time.md](time.md)
@@ -13,6 +22,14 @@ successfully, a following *open* may assign the same id to a new recording.
 The open id disambiguates this and should be used whenever referring to a
 recording that may be unflushed.
 
+*ppm:* Part Per Million.  Crystal Clock accuracy is defined in terms of ppm or 
+parts per million and it gives a convenient way of comparing accuracies 
+of different crystal specifications. "A typical crystal has an error of 
+100ppm (ish) this translates as 100/1e6 or (1e-4)...So the total error on a day 
+is 86400 x 1e-4= 8.64 seconds per day. In a month you would loose 
+30x8.64 = 259 seconds or 4.32 minutes per month." 
+Source: https://www.best-microcontroller-projects.com/ppm.html
+
 *recording:* the video from a (typically 1-minute) portion of an RTSP session.
 RTSP sessions are divided into recordings as a detail of the
 storage schema. See [schema.md](schema.md) for details. This concept is exposed

design/time.md

@@ -111,7 +111,7 @@ information:
     use SNTP clients which simply step time periodically and provide no
     interface to determine if the clock is currently synchronized. This
     document's author owns several cameras with clocks that run roughly 20
-    ppm fast (2 seconds per day) and are adjusted via steps.
+    *ppm* fast (2 seconds per day) and are adjusted via steps.
 *   the RTP timestamps from each of a camera's streams. As described in
     [RFC 3550 section 5.1](https://tools.ietf.org/html/rfc3550#section-5.1),
     these are monotonically increasing with an unspecified reference point.
@@ -158,7 +158,7 @@ relying on the camera's clock for the *media duration* of frames and
 recordings. In the first recording in a *run*, it will use these durations
 and the NVR's wall clock time to establish the start time of the run. In
 subsequent recordings of the run, it will calculate a *wall duration* which
-is up to 500 ppm different from the media duration to gently correct the
+is up to 500 *ppm* different from the media duration to gently correct the
 camera's clock toward the NVR's clock, trusting the latter to be more
 accurate.
 
@@ -207,17 +207,17 @@ a *wall duration* of recordings which more closely matches the NVR's clock.
 It is calculated as follows:
 
 *   For the first recording in a run: the wall duration is the media duration.
-    At the design limit of 500 ppm camera frequency error and an upper
+    At the design limit of 500 *ppm* camera frequency error and an upper
     bound of two minutes duration for the initial recording, this causes
     a maximum of 60 milliseconds of error.
 *   For subsequent recordings, the wall duration is the media duration
-    adjusted by up to 500 ppm to reduce differences between the "local start
+    adjusted by up to 500 *ppm* to reduce differences between the "local start
     time" and the start time, as follows:
     ```
     limit = media_duration / 2000
     wall_duration = media_duration + clamp(local_start - start, -limit, +limit)
     ```
-    Note that for a 1-minute recording, 500 ppm is 0.3 ms, or 27 90kHz units.
+    Note that for a 1-minute recording, 500 *ppm* is 0.3 ms, or 27 90kHz units.
 
 Each recording's local start time is also stored in the database as a delta to
 the recording's start time. These stored values aren't used for normal system
@@ -342,7 +342,7 @@ second would still be ambiguous.
 
 Moonfire NVR could make no code changes and ask the system administrator to
 use smeared time. This is the simplest option. On a leap smear system, there
-are no time jumps. The ~11.6 ppm frequency error and the maximum introduced
+are no time jumps. The ~11.6 *ppm* frequency error and the maximum introduced
 absolute error of 0.5 sec can be considered acceptable.
 
 Alternatively, Moonfire NVR could assume a specific leap smear policy (such as