MyFavMirrors/moonfire-nvr
1
0
Fork 0
mirror of https://github.com/scottlamb/moonfire-nvr.git synced 2025-01-12 15:33:22 -05:00
Code Issues Packages Projects Releases Wiki Activity

mass markdown reformatting

Browse Source 
Add tables of contents (using the VS Code Markdown All-In-One extension)
and reformat lists to consistently use 4-space indents. No content
changes.
This commit is contained in:
Scott Lamb 2021-04-01 12:10:43 -07:00
parent 74b13a0fbf
commit 4d4d78ba64
10 changed files with 539 additions and 470 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
.vscode/settings.json vendored
View File

@ -21,7 +21,7 @@
// I find Prettier's markdown style jarring, including converting `*` // I find Prettier's markdown style jarring, including converting `*`
// bullets to `-` and two-column indents. It's not customizable either. // bullets to `-` and two-column indents. It's not customizable either.
// Don't use it. // Don't use it.
"editor.defaultFormatter": null "editor.defaultFormatter": "yzhang.markdown-all-in-one"
}, },
// Rust-specific overrides. // Rust-specific overrides.
@ -41,5 +41,7 @@
"editor.defaultFormatter": "matklad.rust-analyzer" "editor.defaultFormatter": "matklad.rust-analyzer"
//"editor.defaultFormatter": null //"editor.defaultFormatter": null
}, },
"rust-analyzer.inlayHints.enable": false "rust-analyzer.inlayHints.enable": false,
"markdown.extension.list.indentationSize": "inherit",
"markdown.extension.toc.unorderedList.marker": "*"
} }

24
design/api.md
View File

@ -1,7 +1,27 @@
# Moonfire NVR API # Moonfire NVR API <!-- omit in toc -->
Status: **current**. Status: **current**.
* [Objective](#objective)
* [Detailed design](#detailed-design)
* [`POST /api/login`](#post-apilogin)
* [`POST /api/logout`](#post-apilogout)
* [`GET /api/`](#get-api)
* [`GET /api/cameras/<uuid>/`](#get-apicamerasuuid)
* [`GET /api/cameras/<uuid>/<stream>/recordings`](#get-apicamerasuuidstreamrecordings)
* [`GET /api/cameras/<uuid>/<stream>/view.mp4`](#get-apicamerasuuidstreamviewmp4)
* [`GET /api/cameras/<uuid>/<stream>/view.mp4.txt`](#get-apicamerasuuidstreamviewmp4txt)
* [`GET /api/cameras/<uuid>/<stream>/view.m4s`](#get-apicamerasuuidstreamviewm4s)
* [`GET /api/cameras/<uuid>/<stream>/view.m4s.txt`](#get-apicamerasuuidstreamviewm4stxt)
* [`GET /api/cameras/<uuid>/<stream>/live.m4s`](#get-apicamerasuuidstreamlivem4s)
* [`GET /api/init/<id>.mp4`](#get-apiinitidmp4)
* [`GET /api/init/<id>.mp4.txt`](#get-apiinitidmp4txt)
* [`GET /api/signals`](#get-apisignals)
* [`POST /api/signals`](#post-apisignals)
* [Request 1](#request-1)
* [Request 2](#request-2)
* [Request 3](#request-3)
## Objective ## Objective
Allow a JavaScript-based web interface to list cameras and view recordings. Allow a JavaScript-based web interface to list cameras and view recordings.
@ -704,7 +724,7 @@ Response:
} }
``` ```
### Request 3 #### Request 3
5 seconds later, the client observes motion has ended. It leaves the prior 5 seconds later, the client observes motion has ended. It leaves the prior
data alone and predicts no more motion. data alone and predicts no more motion.

225
design/schema.md
View File

@ -1,4 +1,4 @@
# Moonfire NVR Storage Schema # Moonfire NVR Storage Schema <!-- omit in toc -->
Status: **current**. Status: **current**.
@ -6,6 +6,20 @@ This is the initial design for the most fundamental parts of the Moonfire NVR
storage schema. See also [guide/schema.md](../guide/schema.md) for more storage schema. See also [guide/schema.md](../guide/schema.md) for more
administrator-focused documentation. administrator-focused documentation.
* [Objective](#objective)
* [Cameras](#cameras)
* [Hard drives](#hard-drives)
* [Overview](#overview)
* [Detailed design](#detailed-design)
* [SQLite3](#sqlite3)
* [Duration of recordings](#duration-of-recordings)
* [Lifecycle of a sample file directory](#lifecycle-of-a-sample-file-directory)
* [Lifecycle of a recording](#lifecycle-of-a-recording)
* [Verifying invariants](#verifying-invariants)
* [Recording table](#recording-table)
* [`video_index`](#video_index)
* [<a href="on-demand"></a>On-demand `.mp4` construction](#on-demand-mp4-construction)
## Objective ## Objective
Goals: Goals:
@ -73,13 +87,17 @@ different quality settings as well.
Decode: Decode:
$ time ffmpeg -y -threads 1 -i input.mp4 \ ```
$ time ffmpeg -y -threads 1 -i input.mp4 \
-f null /dev/null -f null /dev/null
```
Combo (Decode + encode with libx264): Combo (Decode + encode with libx264):
$ time ffmpeg -y -threads 1 -i input.mp4 \ ```
$ time ffmpeg -y -threads 1 -i input.mp4 \
-c:v libx264 -preset ultrafast -threads 1 -f mp4 /dev/null -c:v libx264 -preset ultrafast -threads 1 -f mp4 /dev/null
```
| Processor | 1080p30 decode | 1080p30 combo | 704x480p10 decode | 704x480p10 combo | | Processor | 1080p30 decode | 1080p30 combo | 704x480p10 decode | 704x480p10 combo |
@ -115,8 +133,10 @@ only capable of 50 random accesses per second, and each one takes time that
otherwise could be used to transfer 2+ MB. The constrained resource, *disk otherwise could be used to transfer 2+ MB. The constrained resource, *disk
time fraction*, can be bounded as follows: time fraction*, can be bounded as follows:
disk time fraction <= (seek rate) / (50 seeks/sec) + ```
disk time fraction <= (seek rate) / (50 seeks/sec) +
(bandwidth) / (100 MB/sec) (bandwidth) / (100 MB/sec)
```
## Overview ## Overview
@ -131,15 +151,16 @@ Each recording is stored in two places:
Each file in this directory is simply a concatenation of the compressed, Each file in this directory is simply a concatenation of the compressed,
timestamped video samples (also called "packets" or encoded frames), as timestamped video samples (also called "packets" or encoded frames), as
received from the camera. In MPEG-4 terminology (see [ISO received from the camera. In MPEG-4 terminology (see [ISO
14496-12][iso-14496-12]), this is the contents of a `mdat` box for a `.mp4` 14496-12][iso-14496-12]), this is the contents of a `mdat` box for a
file representing the segment. These files do not contain framing data (start `.mp4` file representing the segment. These files do not contain framing
and end byte offsets of samples) and thus are not meant to be decoded on data (start and end byte offsets of samples) and thus are not meant to be
their own. decoded on their own.
* the `recording` table in a [SQLite3][sqlite3] database, intended to be * the `recording` table in a [SQLite3][sqlite3] database, intended to be
stored on flash if possible. A row in this table contains all the metadata stored on flash if possible. A row in this table contains all the
associated with the segment, including the sample-by-sample contents of the metadata associated with the segment, including the sample-by-sample
MPEG-4 `stbl` box. At 30 fps, a row is expected to require roughly 4 KB of contents of the MPEG-4 `stbl` box. At 30 fps, a row is expected to
storage (2 bytes per sample, plus some fixed overhead). require roughly 4 KB of storage (2 bytes per sample, plus some fixed
overhead).
Putting the metadata on flash means metadata operations can be fast Putting the metadata on flash means metadata operations can be fast
(sub-millisecond random access, with parallelism) and do not take precious (sub-millisecond random access, with parallelism) and do not take precious
@ -168,16 +189,17 @@ the SQLite3 database using [write-ahead logging][sqlite3-wal]. There are
several reasons for this decision: several reasons for this decision:
* No user administration required. SQLite3, unlike its heavier-weight friends * No user administration required. SQLite3, unlike its heavier-weight friends
MySQL and PostgreSQL, can be completely internal to the application. In many MySQL and PostgreSQL, can be completely internal to the application. In
applications, end users are unaware of the existence of a RDBMS, and many applications, end users are unaware of the existence of a RDBMS, and
Moonfire NVR should be no exception. Moonfire NVR should be no exception.
* Correctness. It's relatively easy to make guarantees about the state of an * Correctness. It's relatively easy to make guarantees about the state of an
ACID database, and SQLite3 in particular has a robust implementation. (See ACID database, and SQLite3 in particular has a robust implementation.
[Files Are Hard][file-consistency].) (See [Files Are Hard][file-consistency].)
* Developer ease and familiarity. SQL-based RDBMSs are quite common and * Developer ease and familiarity. SQL-based RDBMSs are quite common and
provide a lot of high-level constructs that ease development. SQLite3 in provide a lot of high-level constructs that ease development. SQLite3 in
particular is ubiquitous. Contributors are likely to come with some particular is ubiquitous. Contributors are likely to come with some
understanding of the database, and there are many resources to learn more. understanding of the database, and there are many resources to learn
more.
Total database size is expected to be roughly 4 KB per minute at 30 fps, or Total database size is expected to be roughly 4 KB per minute at 30 fps, or
1 GB for six camera-months of video. This will easily fit on a modest flash 1 GB for six camera-months of video. This will easily fit on a modest flash
@ -190,21 +212,23 @@ There are many constraints that influenced the choice of 1 minute as the
duration of recordings. duration of recordings.
* Per-recording metadata size. There is a fixed component to the size of each * Per-recording metadata size. There is a fixed component to the size of each
row, including the starting/ending timestamps, sample file UUID, etc. This row, including the starting/ending timestamps, sample file UUID, etc.
should not cause the database to be too large to fit on low-cost flash This should not cause the database to be too large to fit on low-cost
devices. As described in the previous section, with 1 minute recordings the flash devices. As described in the previous section, with 1 minute
size is quite modest. recordings the size is quite modest.
* Disk seeks. Sample files should be large enough that even during * Disk seeks. Sample files should be large enough that even during
simultaneous recording and playback of several streams, the disk seeks simultaneous recording and playback of several streams, the disk seeks
incurred when switching from one file to another should not be significant. incurred when switching from one file to another should not be
At the extreme, a sample file per frame could cause an unacceptable 240 significant. At the extreme, a sample file per frame could cause an
seeks per second just to record 8 30 fps streams. At one minute recording unacceptable 240 seeks per second just to record 8 30 fps streams. At one
time, 16 recording streams (2 per each of 8 cameras) and 4 playback streams minute recording time, 16 recording streams (2 per each of 8 cameras) and
would cause on average 20 seeks per minute, or under 1% disk time. 4 playback streams would cause on average 20 seeks per minute, or under
1% disk time.
* Internal fragmentation. Common Linux filesystems have a block size of 4 KiB * Internal fragmentation. Common Linux filesystems have a block size of 4 KiB
(see `statvfs.f_frsize`). Up to this much space per file will be wasted at (see `statvfs.f_frsize`). Up to this much space per file will be wasted
the end of each file. At the bitrates described in "Background", this is an at the end of each file. At the bitrates described in "Background", this
insignicant .02% waste for main streams and .5% waste for sub streams. is an insignicant .02% waste for main streams and .5% waste for sub
streams.
* Number of "slices" in .mp4 files. As described [below](#on-demand), * Number of "slices" in .mp4 files. As described [below](#on-demand),
`.mp4` files will be constructed on-demand for export. It should be `.mp4` files will be constructed on-demand for export. It should be
possible to export an hours-long segment without too much overhead. In possible to export an hours-long segment without too much overhead. In
@ -214,8 +238,8 @@ duration of recordings.
* Crashes. On program crash or power loss, ideally it's acceptable to simply * Crashes. On program crash or power loss, ideally it's acceptable to simply
discard any recordings in progress rather than add a checkpointing scheme. discard any recordings in progress rather than add a checkpointing scheme.
* Granularity of retention. It should be possible to extend retention time * Granularity of retention. It should be possible to extend retention time
around motion events without forcing retention of too much additional data around motion events without forcing retention of too much additional
or copying bytes around on disk. data or copying bytes around on disk.
The design avoids the need for the following constraints: The design avoids the need for the following constraints:
@ -234,13 +258,14 @@ inconsistency. There are many ways this could arise:
failure or misconfiguration. failure or misconfiguration.
* the administrator mixing up the mount points of two filesystems holding * the administrator mixing up the mount points of two filesystems holding
different sample file directories. different sample file directories.
* the administrator renaming a sample file directory without updating the database. * the administrator renaming a sample file directory without updating the
database.
* the administrator restoring the database from backup but not the sample file * the administrator restoring the database from backup but not the sample file
directory, or vice versa. directory, or vice versa.
* the administrator providing two sample file directory paths pointed at the * the administrator providing two sample file directory paths pointed at the
same inode via symlinks or non-canonical paths. (Note that flock(2) has a same inode via symlinks or non-canonical paths. (Note that flock(2) has a
design flaw in which multiple file descriptors can share a lock, so the current design flaw in which multiple file descriptors can share a lock, so the
locking scheme is not sufficient to detect this otherwise.) current locking scheme is not sufficient to detect this otherwise.)
* database and sample file directories forked from the same version, opened * database and sample file directories forked from the same version, opened
the same number of times, then crossed. the same number of times, then crossed.
@ -323,40 +348,40 @@ This is a sub-procedure used in several places below.
Precondition: the directory's lock is held with `LOCK_EX` (exclusive) and Precondition: the directory's lock is held with `LOCK_EX` (exclusive) and
there is an existing metadata file. there is an existing metadata file.
1. Open the metadata file. 1. Open the metadata file.
2. Rewrite the fixed-length data atomically. 2. Rewrite the fixed-length data atomically.
3. `fdatasync` the file. 3. `fdatasync` the file.
*Open the database as read-only* *Open the database as read-only*
1. Lock the database directory with `LOCK_SH` (shared). 1. Lock the database directory with `LOCK_SH` (shared).
2. Open the SQLite database with `SQLITE_OPEN_READ_ONLY`. 2. Open the SQLite database with `SQLITE_OPEN_READ_ONLY`.
*Open the database as read-write* *Open the database as read-write*
1. Lock the database directory with `LOCK_EX` (exclusive). 1. Lock the database directory with `LOCK_EX` (exclusive).
2. Open the SQLite database with `SQLITE_OPEN_READ_WRITE`. 2. Open the SQLite database with `SQLITE_OPEN_READ_WRITE`.
3. Insert a new `open` table row with the new sequence number and uuid. 3. Insert a new `open` table row with the new sequence number and uuid.
*Create a sample file directory* *Create a sample file directory*
Precondition: database open read-write. Precondition: database open read-write.
1. Lock the sample file directory with `LOCK_EX` (exclusive). 1. Lock the sample file directory with `LOCK_EX` (exclusive).
2. Verify there is no metadata file or `last_complete_open` is unset. 2. Verify there is no metadata file or `last_complete_open` is unset.
3. Write new metadata file with a fresh `dir_uuid` and a `in_progress_open` 3. Write new metadata file with a fresh `dir_uuid` and a `in_progress_open`
matching the database's current open. matching the database's current open.
4. Add a matching row to the database with `last_complete_open_id` matching 4. Add a matching row to the database with `last_complete_open_id` matching
the current open. the current open.
5. Update the metadata file to move `in_progress_open` to 5. Update the metadata file to move `in_progress_open` to
`last_complete_open`. `last_complete_open`.
*Open a sample file directory read-only* *Open a sample file directory read-only*
Precondition: database open (read-only or read-write). Precondition: database open (read-only or read-write).
1. Lock the sample file directory with `LOCK_SH` (shared). 1. Lock the sample file directory with `LOCK_SH` (shared).
2. Verify the metadata file matches the database: 2. Verify the metadata file matches the database:
* database uuid matches. * database uuid matches.
* dir uuid matches. * dir uuid matches.
* if the database's `last_complete_open` is set, it must match the * if the database's `last_complete_open` is set, it must match the
@ -368,29 +393,29 @@ Precondition: database open (read-only or read-write).
Precondition: database open read-write. Precondition: database open read-write.
1. Lock the sample file directory with `LOCK_EX` (exclusive). 1. Lock the sample file directory with `LOCK_EX` (exclusive).
2. Verify the metadata file matches the database (as above). 2. Verify the metadata file matches the database (as above).
3. Update the metadata file with `in_progress_open` matching the current 3. Update the metadata file with `in_progress_open` matching the current
open. open.
3. Update the database row with `last_complete_open_id` matching the current 4. Update the database row with `last_complete_open_id` matching the current
open. open.
4. Update the metadata file with `last_complete_open` rather than 5. Update the metadata file with `last_complete_open` rather than
`in_progress_open`. `in_progress_open`.
5. Run the recording startup procedure for this directory. 6. Run the recording startup procedure for this directory.
*Close a sample file directory* *Close a sample file directory*
1. Drop the sample file directory lock. 1. Drop the sample file directory lock.
*Delete a sample file directory* *Delete a sample file directory*
1. Remove all sample files (of all three categories described below: 1. Remove all sample files (of all three categories described below:
`recording` table rows, `garbage` table rows, and files with recording `recording` table rows, `garbage` table rows, and files with recording
ids >= their stream's `cum_recordings`); see "delete a recording" ids >= their stream's `cum_recordings`); see "delete a recording"
procedure below. procedure below.
2. Rewrite the directory metadata with `in_progress_open` set to the current open, 2. Rewrite the directory metadata with `in_progress_open` set to the current open,
`last_complete_open` cleared. `last_complete_open` cleared.
3. Delete the directory's row from the database. 3. Delete the directory's row from the database.
### Lifecycle of a recording ### Lifecycle of a recording
@ -398,14 +423,14 @@ Because a major part of the recording state is outside the SQL database, care
must be taken to guarantee consistency and durability. Moonfire NVR maintains must be taken to guarantee consistency and durability. Moonfire NVR maintains
three invariants about sample files: three invariants about sample files:
1. `recording` table rows have sample files on disk with the indicated size 1. `recording` table rows have sample files on disk with the indicated size
and SHA-1 hash. and SHA-1 hash.
2. Exactly one of the following statements is true for every sample file: 2. Exactly one of the following statements is true for every sample file:
* It has a `recording` table row. * It has a `recording` table row.
* It has a `garbage` table row. * It has a `garbage` table row.
* Its recording id is greater than or equal to the `cum_recordings` * Its recording id is greater than or equal to the `cum_recordings`
for its stream. for its stream.
3. After an orderly shutdown of Moonfire NVR, there is a `recording` table row 3. After an orderly shutdown of Moonfire NVR, there is a `recording` table row
for every sample file, even if there have been previous crashes. for every sample file, even if there have been previous crashes.
The first invariant provides certainty that a recording is properly stored. It The first invariant provides certainty that a recording is properly stored. It
@ -487,15 +512,17 @@ The times are roughly:
The `readdir()` and `fstat()` times can be tested simply: The `readdir()` and `fstat()` times can be tested simply:
$ mkdir testdir ```
$ cd testdir $ mkdir testdir
$ seq 1 $[60*24*365*6/12*2] | xargs touch $ cd testdir
$ sudo sh -c 'echo 1 > /proc/sys/vm/drop_caches' $ seq 1 $[60*24*365*6/12*2] | xargs touch
$ time ls -1 -f | wc -l $ sudo sh -c 'echo 1 > /proc/sys/vm/drop_caches'
$ sudo sh -c 'echo 1 > /proc/sys/vm/drop_caches' $ time ls -1 -f | wc -l
$ time ls -1 -f --size | wc -l $ sudo sh -c 'echo 1 > /proc/sys/vm/drop_caches'
$ time ls -1 -f --size | wc -l
```
(The system calls used by `ls` can be verified through strace.) (The system calls used by `ls` can be verified through strace.)
The hash verification time is easiest to calculate: reading 5.9 TB at 100 The hash verification time is easiest to calculate: reading 5.9 TB at 100
MB/sec takes about 8 hours. On some systems, it will be even slower. On MB/sec takes about 8 hours. On some systems, it will be even slower. On
@ -515,8 +542,9 @@ the background at low priority.
The snippet below is a illustrative excerpt of the SQLite schema; see The snippet below is a illustrative excerpt of the SQLite schema; see
`schema.sql` for the authoritative, up-to-date version. `schema.sql` for the authoritative, up-to-date version.
-- A single, typically 60-second, recorded segment of video. ```sql
create table recording ( -- A single, typically 60-second, recorded segment of video.
create table recording (
id integer primary key, id integer primary key,
open_id integer references open (id), open_id integer references open (id),
camera_id integer references camera (id) not null, camera_id integer references camera (id) not null,
@ -535,11 +563,11 @@ The snippet below is a illustrative excerpt of the SQLite schema; see
video_index blob, video_index blob,
... ...
); );
-- A concrete box derived from a ISO/IEC 14496-12 section 8.5.2 -- A concrete box derived from a ISO/IEC 14496-12 section 8.5.2
-- VisualSampleEntry box. Describes the codec, width, height, etc. -- VisualSampleEntry box. Describes the codec, width, height, etc.
create table visual_sample_entry ( create table visual_sample_entry (
id integerprimary key, id integerprimary key,
-- The width and height in pixels; must match values within -- The width and height in pixels; must match values within
@ -550,7 +578,8 @@ The snippet below is a illustrative excerpt of the SQLite schema; see
-- A serialized SampleEntry box, including the leading length and box -- A serialized SampleEntry box, including the leading length and box
-- type (avcC in the case of H.264). -- type (avcC in the case of H.264).
data blob data blob
); );
```
As mentioned by the `start_time_90k` field above, recordings use a 90 kHz time As mentioned by the `start_time_90k` field above, recordings use a 90 kHz time
base. This matches the RTP timestamp frequency used for H.264 and other video base. This matches the RTP timestamp frequency used for H.264 and other video
@ -614,16 +643,18 @@ This encoding is chosen so that values will be near zero, and thus the varints
will be at their most compact possible form. An index might be written by the will be at their most compact possible form. An index might be written by the
following pseudocode: following pseudocode:
prev_duration = 0 ```
prev_bytes_key = 0 prev_duration = 0
prev_bytes_nonkey = 0 prev_bytes_key = 0
for each frame: prev_bytes_nonkey = 0
for each frame:
duration_delta = duration - prev_duration duration_delta = duration - prev_duration
bytes_delta = bytes - (is_key ? prev_bytes_key : prev_bytes_nonkey) bytes_delta = bytes - (is_key ? prev_bytes_key : prev_bytes_nonkey)
prev_duration_ms = duration_ms prev_duration_ms = duration_ms
if key: prev_bytes_key = bytes else: prev_bytes_nonkey = bytes if key: prev_bytes_key = bytes else: prev_bytes_nonkey = bytes
PutVarint((Zigzag(duration_delta) << 1) | is_key) PutVarint((Zigzag(duration_delta) << 1) | is_key)
PutVarint(Zigzag(bytes_delta) PutVarint(Zigzag(bytes_delta)
```
See also the example below: See also the example below:
@ -654,38 +685,6 @@ from several segments. The implementation details are outside the scope of this
document, but this is possible in part due to the use of an on-flash database document, but this is possible in part due to the use of an on-flash database
to store metadata and the simple, consistent format of sample indexes. to store metadata and the simple, consistent format of sample indexes.
### Copyright
This file is part of Moonfire NVR, a security camera network video recorder.
Copyright (C) 2016 The Moonfire NVR Authors
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
In addition, as a special exception, the copyright holders give
permission to link the code of portions of this program with the
OpenSSL library under certain conditions as described in each
individual source file, and distribute linked combinations including
the two.
You must obey the GNU General Public License in all respects for all
of the code used other than OpenSSL. If you modify file(s) with this
exception, you may extend this exception to your version of the
file(s), but you are not obligated to do so. If you do not wish to do
so, delete this exception statement from your version. If you delete
this exception statement from all source files in the program, then
also delete it here.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
[pi2]: https://www.raspberrypi.org/products/raspberry-pi-2-model-b/ [pi2]: https://www.raspberrypi.org/products/raspberry-pi-2-model-b/
[xandem]: http://www.xandemhome.com/ [xandem]: http://www.xandemhome.com/
[hikcam]: http://overseas.hikvision.com/us/Products_accessries_10533_i7696.html [hikcam]: http://overseas.hikvision.com/us/Products_accessries_10533_i7696.html

71
design/time.md
View File

@ -1,4 +1,4 @@
# Moonfire NVR Time Handling # Moonfire NVR Time Handling <!-- omit in toc -->
Status: **current**. Status: **current**.
@ -7,6 +7,19 @@ Status: **current**.
> >
> — Segal's law > — Segal's law
* [Objective](#objective)
* [Background](#background)
* [Overview](#overview)
* [Detailed design](#detailed-design)
* [Caveats](#caveats)
* [Stream mismatches](#stream-mismatches)
* [Time discontinuities](#time-discontinuities)
* [Leap seconds](#leap-seconds)
* [Use `clock_gettime(CLOCK_TAI, ...)` timestamps](#use-clock_gettimeclock_tai--timestamps)
* [Use a leap second table when calculating differences](#use-a-leap-second-table-when-calculating-differences)
* [Use smeared time](#use-smeared-time)
* [Alternatives considered](#alternatives-considered)
## Objective ## Objective
Maximize the likelihood Moonfire NVR's timestamps are useful. Maximize the likelihood Moonfire NVR's timestamps are useful.
@ -14,18 +27,18 @@ Maximize the likelihood Moonfire NVR's timestamps are useful.
The timestamp corresponding to a video frame should roughly match timestamps The timestamp corresponding to a video frame should roughly match timestamps
from other sources: from other sources:
* another video stream from the same camera. Given a video frame from the * another video stream from the same camera. Given a video frame from the
"main" stream, a video frame from the "sub" stream with a similar "main" stream, a video frame from the "sub" stream with a similar
timestamp should have been recorded near the same time, and vice versa. timestamp should have been recorded near the same time, and vice versa.
This minimizes confusion when switching between views of these streams, This minimizes confusion when switching between views of these streams,
and when viewing the "main" stream timestamps corresponding to a motion and when viewing the "main" stream timestamps corresponding to a motion
event gathered from the less CPU-intensive "sub" stream. event gathered from the less CPU-intensive "sub" stream.
* on-camera motion events from the same camera. If the video frame reflects * on-camera motion events from the same camera. If the video frame reflects
the motion event, its timestamp should be roughly within the event's the motion event, its timestamp should be roughly within the event's
timespan. timespan.
* streams from other cameras. Recorded views from two cameras of the same * streams from other cameras. Recorded views from two cameras of the same
event should have similar timestamps. event should have similar timestamps.
* events noted by the owner of the system, neighbors, police, etc., for the * events noted by the owner of the system, neighbors, police, etc., for the
purpose of determining chronology, to the extent those persons use purpose of determining chronology, to the extent those persons use
accurate clocks. accurate clocks.
@ -35,26 +48,26 @@ recorded video.
Durations should be useful over short timescales: Durations should be useful over short timescales:
* If an object's motion is recorded, distance travelled divided by the * If an object's motion is recorded, distance travelled divided by the
duration of the frames over which this motion occurred should reflect the duration of the frames over which this motion occurred should reflect the
object's average speed. object's average speed.
* Motion should appear smooth. There shouldn't be excessive frame-to-frame * Motion should appear smooth. There shouldn't be excessive frame-to-frame
jitter due to such factors as differences in encoding time or network jitter due to such factors as differences in encoding time or network
transmission. transmission.
This document describes an approach to achieving these goals when the This document describes an approach to achieving these goals when the
following statements are true: following statements are true:
* the NVR's system clock is within a second of correct on startup. (True * the NVR's system clock is within a second of correct on startup. (True
when NTP is functioning or when the system has a real-time clock battery when NTP is functioning or when the system has a real-time clock battery
to preserve a previous correct time.) to preserve a previous correct time.)
* the NVR's system time does not experience forward or backward "step" * the NVR's system time does not experience forward or backward "step"
corrections (as opposed to frequency correction) during operation. corrections (as opposed to frequency correction) during operation.
* the NVR's system time advances at roughly the correct frequency. (NTP * the NVR's system time advances at roughly the correct frequency. (NTP
achieves this through frequency correction when operating correctly.) achieves this through frequency correction when operating correctly.)
* the cameras' clock frequencies are off by no more than 500 parts per * the cameras' clock frequencies are off by no more than 500 parts per
million (roughly 43 seconds per day). million (roughly 43 seconds per day).
* the cameras are geographically close to the NVR, so in most cases network * the cameras are geographically close to the NVR, so in most cases network
transmission time is under 50 ms. (Occasional delays are to be expected, transmission time is under 50 ms. (Occasional delays are to be expected,
however.) however.)
@ -81,31 +94,31 @@ so such problems are to be expected.
Moonfire NVR typically has access to the following sources of time Moonfire NVR typically has access to the following sources of time
information: information:
* the local `CLOCK_REALTIME`. Ideally this is maintained by `ntpd`: * the local `CLOCK_REALTIME`. Ideally this is maintained by `ntpd`:
synchronized on startup, and frequency-corrected during operation. A synchronized on startup, and frequency-corrected during operation. A
hardware real-time clock and battery keep accurate time across restarts hardware real-time clock and battery keep accurate time across restarts
if the network is unavailable on startup. In the worst case, the system if the network is unavailable on startup. In the worst case, the system
has no real-time clock or no battery and a network connection is has no real-time clock or no battery and a network connection is
unavailable. The time is far in the past on startup and is never unavailable. The time is far in the past on startup and is never
corrected or is corrected via a step while Moonfire NVR is running. corrected or is corrected via a step while Moonfire NVR is running.
* the local `CLOCK_MONOTONIC`. This should be frequency-corrected by `ntpd` * the local `CLOCK_MONOTONIC`. This should be frequency-corrected by `ntpd`
and guaranteed to never experience "steps", though its reference point is and guaranteed to never experience "steps", though its reference point is
unspecified. unspecified.
* the local `ntpd`, which can be used to determine if the system is * the local `ntpd`, which can be used to determine if the system is
synchronized to NTP and quantify the precision of synchronization. synchronized to NTP and quantify the precision of synchronization.
* each camera's clock. The ONVIF specification mandates cameras must * each camera's clock. The ONVIF specification mandates cameras must
support synchronizing clocks via NTP, but in practice cameras appear to support synchronizing clocks via NTP, but in practice cameras appear to
use SNTP clients which simply step time periodically and provide no use SNTP clients which simply step time periodically and provide no
interface to determine if the clock is currently synchronized. This interface to determine if the clock is currently synchronized. This
document's author owns several cameras with clocks that run roughly 20 document's author owns several cameras with clocks that run roughly 20
ppm fast (2 seconds per day) and are adjusted via steps. ppm fast (2 seconds per day) and are adjusted via steps.
* the RTP timestamps from each of a camera's streams. As described in * the RTP timestamps from each of a camera's streams. As described in
[RFC 3550 section 5.1](https://tools.ietf.org/html/rfc3550#section-5.1), [RFC 3550 section 5.1](https://tools.ietf.org/html/rfc3550#section-5.1),
these are monotonically increasing with an unspecified reference point. these are monotonically increasing with an unspecified reference point.
They can't be directly compared to other cameras or other streams from They can't be directly compared to other cameras or other streams from
the same camera. Emperically, budget cameras don't appear to do any the same camera. Emperically, budget cameras don't appear to do any
frequency correction on these timestamps. frequency correction on these timestamps.
* in some cases, RTCP sender reports, as described in * in some cases, RTCP sender reports, as described in
[RFC 3550 section 6.4](https://tools.ietf.org/html/rfc3550#section-6.4). [RFC 3550 section 6.4](https://tools.ietf.org/html/rfc3550#section-6.4).
These correlate RTP timestamps with the camera's real time clock. These correlate RTP timestamps with the camera's real time clock.
However, these are only sent periodically, not necessarily at the However, these are only sent periodically, not necessarily at the
@ -224,14 +237,14 @@ wall clock, and thus calculate the camera's time as of the first frame.
The _start time_ of the first recording could be either its local start time The _start time_ of the first recording could be either its local start time
or its camera start time, determined via the following rules: or its camera start time, determined via the following rules:
1. if there is no camera start time (due to the lack of a RTCP sender 1. if there is no camera start time (due to the lack of a RTCP sender
report), the local start time wins by default. report), the local start time wins by default.
2. if the camera start time is before 2016-01-01 00:00:00 UTC, the local 2. if the camera start time is before 2016-01-01 00:00:00 UTC, the local
start time wins. start time wins.
3. if the local start time is before 2016-01-01 00:00:00 UTC, the camera 3. if the local start time is before 2016-01-01 00:00:00 UTC, the camera
start time wins. start time wins.
4. if the times differ by more than 5 seconds, the local start time wins. 4. if the times differ by more than 5 seconds, the local start time wins.
5. otherwise, the camera start time wins. 5. otherwise, the camera start time wins.
These rules are a compromise. When a system starts up without NTP or a clock These rules are a compromise. When a system starts up without NTP or a clock
battery, it typically reverts to a time in the distant past. Therefore times battery, it typically reverts to a time in the distant past. Therefore times
@ -259,9 +272,9 @@ happened](https://github.com/scottlamb/moonfire-nvr/issues/9#issuecomment-322663
Moonfire NVR will continue to use the initial wall clock time for as long as Moonfire NVR will continue to use the initial wall clock time for as long as
the recording lasts. This can result in some unfortunate behaviors: the recording lasts. This can result in some unfortunate behaviors:
* a recording that lasts for months might have an incorrect time all the * a recording that lasts for months might have an incorrect time all the
way through because `ntpd` took a few minutes on startup. way through because `ntpd` took a few minutes on startup.
* two recordings that were in fact simultaneous might be recorded with very * two recordings that were in fact simultaneous might be recorded with very
different times because a time jump happened between their starts. different times because a time jump happened between their starts.
It might be better to use the new time (assuming that ntpd has made a It might be better to use the new time (assuming that ntpd has made a
@ -299,16 +312,16 @@ Timestamps in the TAI clock system don't skip leap seconds. There's a system
interface intended to provide timestamps in this clock system, and Moonfire interface intended to provide timestamps in this clock system, and Moonfire
NVR could use it. Unfortunately this has several problems: NVR could use it. Unfortunately this has several problems:
* `CLOCK_TAI` is only available on Linux. It'd be preferable to handle * `CLOCK_TAI` is only available on Linux. It'd be preferable to handle
timestamps in a consistent way on other platforms. (At least on macOS, timestamps in a consistent way on other platforms. (At least on macOS,
Moonfire NVR's current primary development platform.) Moonfire NVR's current primary development platform.)
* `CLOCK_TAI` is wrong on startup and possibly adjusted later. The offset * `CLOCK_TAI` is wrong on startup and possibly adjusted later. The offset
between TAI and UTC is initially assumed to be 0. It's corrected when/if between TAI and UTC is initially assumed to be 0. It's corrected when/if
a sufficiently new `ntpd` starts. a sufficiently new `ntpd` starts.
* We'd need a leap second table to translate this into calendar time. One * We'd need a leap second table to translate this into calendar time. One
would have to be downloaded from the Internet periodically, and we'd need would have to be downloaded from the Internet periodically, and we'd need
to consider the case in which the available table is expired. to consider the case in which the available table is expired.
* `CLOCK_TAI` likely doesn't work properly with leap smear systems. Where * `CLOCK_TAI` likely doesn't work properly with leap smear systems. Where
the leap smear prevents a time jump for `CLOCK_REALTIME`, it likely the leap smear prevents a time jump for `CLOCK_REALTIME`, it likely
introduces one for `CLOCK_TAI`. introduces one for `CLOCK_TAI`.

13
guide/build.md
View File

@ -1,4 +1,4 @@
# Building Moonfire NVR # Building Moonfire NVR <!-- omit in toc -->
This document has notes for software developers on building Moonfire NVR from This document has notes for software developers on building Moonfire NVR from
source code for development. If you just want to install precompiled source code for development. If you just want to install precompiled
@ -10,11 +10,10 @@ tracker](https://github.com/scottlamb/moonfire-nvr/issues) or
[mailing list](https://groups.google.com/d/forum/moonfire-nvr-users) when [mailing list](https://groups.google.com/d/forum/moonfire-nvr-users) when
stuck. Please also send pull requests to improve this doc. stuck. Please also send pull requests to improve this doc.
* [Building Moonfire NVR](#building-moonfire-nvr) * [Downloading](#downloading)
* [Downloading](#downloading) * [Docker builds](#docker-builds)
* [Docker builds](#docker-builds)
* [Release procedure](#release-procedure) * [Release procedure](#release-procedure)
* [Non-Docker setup](#non-docker-setup) * [Non-Docker setup](#non-docker-setup)
* [Running interactively straight from the working copy](#running-interactively-straight-from-the-working-copy) * [Running interactively straight from the working copy](#running-interactively-straight-from-the-working-copy)
* [Running as a `systemd` service](#running-as-a-systemd-service) * [Running as a `systemd` service](#running-as-a-systemd-service)
@ -152,8 +151,8 @@ Linux VM and filesystem overlay.
To build the server, you will need the following C libraries installed: To build the server, you will need the following C libraries installed:
* [ffmpeg](http://ffmpeg.org/) version 2.x or 3.x, including `libavutil`, * [ffmpeg](http://ffmpeg.org/) version 2.x or 3.x, including `libavutil`,
`libavcodec` (to inspect H.264 frames), and `libavformat` (to connect to RTSP `libavcodec` (to inspect H.264 frames), and `libavformat` (to connect to
servers and write `.mp4` files). RTSP servers and write `.mp4` files).
Note ffmpeg library versions older than 55.1.101, along with all versions of Note ffmpeg library versions older than 55.1.101, along with all versions of
the competing project [libav](http://libav.org), don't support socket the competing project [libav](http://libav.org), don't support socket

6
guide/developing-ui.md
View File

@ -1,4 +1,8 @@
# Working on UI development # Working on UI development <!-- omit in toc -->
* [Getting started](#getting-started)
* [Overriding defaults](#overriding-defaults)
* [A note on `https`](#a-note-on-https)
The UI is presented from a single HTML page (index.html) and any number The UI is presented from a single HTML page (index.html) and any number
of Javascript files, css files, images, etc. These are "packed" together of Javascript files, css files, images, etc. These are "packed" together

35
guide/install.md
View File

@ -1,4 +1,11 @@
# Downloading, installing, and configuring Moonfire NVR with Docker # Installing Moonfire NVR <!-- omit in toc -->
* [Downloading, installing, and configuring Moonfire NVR with Docker](#downloading-installing-and-configuring-moonfire-nvr-with-docker)
* [Dedicated hard drive setup](#dedicated-hard-drive-setup)
* [Completing configuration through the UI](#completing-configuration-through-the-ui)
* [Starting it up](#starting-it-up)
## Downloading, installing, and configuring Moonfire NVR with Docker
This document describes how to download, install, and configure Moonfire NVR This document describes how to download, install, and configure Moonfire NVR
via the prebuilt Docker images available for x86-64, arm64, and arm. If you via the prebuilt Docker images available for x86-64, arm64, and arm. If you
@ -102,7 +109,7 @@ $ nvr init
This will create a directory `/var/lib/moonfire-nvr/db` with a SQLite3 database This will create a directory `/var/lib/moonfire-nvr/db` with a SQLite3 database
within it. within it.
## Dedicated hard drive setup ### Dedicated hard drive setup
If a dedicated hard drive is available, set up the mount point: If a dedicated hard drive is available, set up the mount point:
@ -139,7 +146,7 @@ mount lines. It will look similar to this:
--mount=type=bind,source=/media/nvr/sample,destination=/media/nvr/sample --mount=type=bind,source=/media/nvr/sample,destination=/media/nvr/sample
``` ```
## Completing configuration through the UI ### Completing configuration through the UI
Once your system is set up, it's time to initialize an empty database Once your system is set up, it's time to initialize an empty database
and add the cameras and sample directories. You can do this and add the cameras and sample directories. You can do this
@ -171,16 +178,16 @@ In the user interface,
* `flush_if_sec` should typically be 120 seconds. This causes the database to * `flush_if_sec` should typically be 120 seconds. This causes the database to
be flushed when the first instant of one of this stream's completed be flushed when the first instant of one of this stream's completed
recordings is 2 minutes old. A "recording" is a segment of a video recordings is 2 minutes old. A "recording" is a segment of a video
stream that is 60–120 seconds when first establishing the stream, about stream that is 60–120 seconds when first establishing the stream,
60 seconds midstream, and shorter when an error or server shutdown about 60 seconds midstream, and shorter when an error or server
terminates the stream. Thus, a value just below 60 will cause the shutdown terminates the stream. Thus, a value just below 60 will
database to be flushed once per minute per stream in the steady state. A cause the database to be flushed once per minute per stream in the
value around 180 will cause the database to be once every 3 minutes per steady state. A value around 180 will cause the database to be once
stream, or less frequently if other streams cause flushes first. Lower every 3 minutes per stream, or less frequently if other streams cause
values cause less video to be lost on power loss. Higher values reduce flushes first. Lower values cause less video to be lost on power
wear on the SSD holding the SQLite database, particularly when you have loss. Higher values reduce wear on the SSD holding the SQLite
many cameras and when you record both the "main" and "sub" streams of database, particularly when you have many cameras and when you record
each camera. both the "main" and "sub" streams of each camera.
3. Assign disk space to your cameras back in "Directories and retention". 3. Assign disk space to your cameras back in "Directories and retention".
Leave a little slack between the total limit and the filesystem capacity, Leave a little slack between the total limit and the filesystem capacity,
@ -202,7 +209,7 @@ In the user interface,
4. Add a user for yourself (and optionally others) under "Users". You'll need 4. Add a user for yourself (and optionally others) under "Users". You'll need
this to access the web UI once you enable authentication. this to access the web UI once you enable authentication.
## Starting it up ### Starting it up
Note that at this stage, Moonfire NVR's web interface is **insecure**: it Note that at this stage, Moonfire NVR's web interface is **insecure**: it
doesn't use `https` and doesn't require you to authenticate doesn't use `https` and doesn't require you to authenticate

40
guide/schema.md
View File

@ -1,4 +1,12 @@
# Moonfire NVR Schema Guide # Moonfire NVR Schema Guide <!-- omit in toc -->
* [Upgrading](#upgrading)
* [Procedure](#procedure)
* [Unversioned to version 0](#unversioned-to-version-0)
* [Version 0 to version 1](#version-0-to-version-1)
* [Version 1 to version 2 to version 3](#version-1-to-version-2-to-version-3)
* [Version 3 to version 4 to version 5](#version-3-to-version-4-to-version-5)
* [Version 6](#version-6)
This document has notes about the Moonfire NVR storage schema. As described in This document has notes about the Moonfire NVR storage schema. As described in
[README.md](../README.md), this consists of two kinds of state: [README.md](../README.md), this consists of two kinds of state:
@ -26,12 +34,12 @@ read-only mode prior to deleting the old database.
First ensure there is sufficient space available for four copies of the First ensure there is sufficient space available for four copies of the
SQLite database: SQLite database:
* copy 1: the copy to upgrade * copy 1: the copy to upgrade
* copy 2: a backup you manually create so that you can restore if you * copy 2: a backup you manually create so that you can restore if you
discover a problem while running the new software against the upgraded discover a problem while running the new software against the upgraded
database in read-only mode. If disk space is tight, you can save this database in read-only mode. If disk space is tight, you can save this
to a different filesystem than the primary copy. to a different filesystem than the primary copy.
* copies 3 and 4: internal copies made and destroyed by Moonfire NVR and * copies 3 and 4: internal copies made and destroyed by Moonfire NVR and
SQLite during the upgrade: SQLite during the upgrade:
* during earlier steps, possibly duplicate copies of tables, which * during earlier steps, possibly duplicate copies of tables, which
@ -56,12 +64,16 @@ Next ensure Moonfire NVR is not running and does not automatically restart if
the system is rebooted during the upgrade. If you followed the Docker the system is rebooted during the upgrade. If you followed the Docker
instructions, you can do this as follows: instructions, you can do this as follows:
$ nvr stop ```
$ nvr stop
```
Then back up your SQLite database. If you are using the default path, you can Then back up your SQLite database. If you are using the default path, you can
do so as follows: do so as follows:
$ sudo -u moonfire-nvr cp /var/lib/moonfire-nvr/db/db{,.pre-upgrade} ```
$ sudo -u moonfire-nvr cp /var/lib/moonfire-nvr/db/db{,.pre-upgrade}
```
By default, the upgrade command will reset the SQLite `journal_mode` to By default, the upgrade command will reset the SQLite `journal_mode` to
`delete` prior to the upgrade. This works around a problem with `delete` prior to the upgrade. This works around a problem with
@ -112,16 +124,16 @@ $ nvr run
Hopefully your system is functioning correctly. If not, there are two options Hopefully your system is functioning correctly. If not, there are two options
for restore; neither are easy: for restore; neither are easy:
* go back to your old database. There will be two classes of problems: * go back to your old database. There will be two classes of problems:
* If the new system deleted any recordings, the old system will * If the new system deleted any recordings, the old system will
incorrectly believe they are still present. You could wait until all incorrectly believe they are still present. You could wait until all
existing files are rotated away, or you could try to delete them existing files are rotated away, or you could try to delete them
manually from the database. manually from the database.
* if the new system created any recordings, the old system will not * If the new system created any recordings, the old system will not
know about them and will not delete them. Your disk may become full. know about them and will not delete them. Your disk may become full.
You should find some way to discover these files and manually delete You should find some way to discover these files and manually delete
them. them.
* undo the changes by hand. There's no documentation on this; you'll need * undo the changes by hand. There's no documentation on this; you'll need
to read the code and come up with a reverse transformation. to read the code and come up with a reverse transformation.
The `nvr check` command will show you what problems exist on your system. The `nvr check` command will show you what problems exist on your system.
@ -136,8 +148,8 @@ will also accept a version 0 database.
Version 0 makes two changes: Version 0 makes two changes:
* it adds schema versioning, as described above. * it adds schema versioning, as described above.
* it adds a column (`video_sync_samples`) to a database index to speed up * it adds a column (`video_sync_samples`) to a database index to speed up
certain operations. certain operations.
There's a special procedure for this upgrade. The good news is that a backup There's a special procedure for this upgrade. The good news is that a backup
@ -150,8 +162,10 @@ Then use `sqlite3` to manually edit the database. The default
path is `/var/lib/moonfire-nvr/db/db`; if you've specified a different path is `/var/lib/moonfire-nvr/db/db`; if you've specified a different
`--db_dir`, use that directory with a suffix of `/db`. `--db_dir`, use that directory with a suffix of `/db`.
$ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db ```
sqlite3> $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db
sqlite3>
```
At the prompt, run the following commands: At the prompt, run the following commands:

14
guide/secure.md
View File

@ -1,4 +1,16 @@
# Securing Moonfire NVR and exposing it to the Internet # Securing Moonfire NVR and exposing it to the Internet <!-- omit in toc -->
* [The problem](#the-problem)
* [VPN or port forwarding?](#vpn-or-port-forwarding)
* [Overview](#overview)
* [1. Install a webserver](#1-install-a-webserver)
* [2. Configure a static internal IP](#2-configure-a-static-internal-ip)
* [3. Set up port forwarding](#3-set-up-port-forwarding)
* [4. Configure a public DNS name](#4-configure-a-public-dns-name)
* [5. Install a TLS certificate](#5-install-a-tls-certificate)
* [6. Reconfigure Moonfire NVR](#6-reconfigure-moonfire-nvr)
* [7. Configure the webserver](#7-configure-the-webserver)
* [Verify it works](#verify-it-works)
## The problem ## The problem

7
guide/troubleshooting.md
View File

@ -1,16 +1,15 @@
# Troubleshooting # Troubleshooting <!-- omit in toc -->
Here are some tips for diagnosing various problems with Moonfire NVR. Feel free Here are some tips for diagnosing various problems with Moonfire NVR. Feel free
to open an [issue](https://github.com/scottlamb/moonfire-nvr/issues) if you to open an [issue](https://github.com/scottlamb/moonfire-nvr/issues) if you
need more help. need more help.
* [Troubleshooting](#troubleshooting) * [Viewing Moonfire NVR's logs](#viewing-moonfire-nvrs-logs)
* [Viewing Moonfire NVR's logs](#viewing-moonfire-nvrs-logs)
* [Flushes](#flushes) * [Flushes](#flushes)
* [Panic errors](#panic-errors) * [Panic errors](#panic-errors)
* [Slow operations](#slow-operations) * [Slow operations](#slow-operations)
* [Camera stream errors](#camera-stream-errors) * [Camera stream errors](#camera-stream-errors)
* [Problems](#problems) * [Problems](#problems)
* [Server errors](#server-errors) * [Server errors](#server-errors)
* [`Error: pts not monotonically increasing; got 26615520 then 26539470`](#error-pts-not-monotonically-increasing-got-26615520-then-26539470) * [`Error: pts not monotonically increasing; got 26615520 then 26539470`](#error-pts-not-monotonically-increasing-got-26615520-then-26539470)
* [Out of disk space](#out-of-disk-space) * [Out of disk space](#out-of-disk-space)