mass markdown reformatting

Add tables of contents (using the VS Code Markdown All-In-One extension)
and reformat lists to consistently use 4-space indents. No content
changes.
This commit is contained in:
Scott Lamb 2021-04-01 12:10:43 -07:00
parent 74b13a0fbf
commit 4d4d78ba64
10 changed files with 539 additions and 470 deletions

View File

@ -21,7 +21,7 @@
// I find Prettier's markdown style jarring, including converting `*`
// bullets to `-` and two-column indents. It's not customizable either.
// Don't use it.
"editor.defaultFormatter": null
"editor.defaultFormatter": "yzhang.markdown-all-in-one"
},
// Rust-specific overrides.
@ -41,5 +41,7 @@
"editor.defaultFormatter": "matklad.rust-analyzer"
//"editor.defaultFormatter": null
},
"rust-analyzer.inlayHints.enable": false
"rust-analyzer.inlayHints.enable": false,
"markdown.extension.list.indentationSize": "inherit",
"markdown.extension.toc.unorderedList.marker": "*"
}

View File

@ -1,7 +1,27 @@
# Moonfire NVR API
# Moonfire NVR API <!-- omit in toc -->
Status: **current**.
* [Objective](#objective)
* [Detailed design](#detailed-design)
* [`POST /api/login`](#post-apilogin)
* [`POST /api/logout`](#post-apilogout)
* [`GET /api/`](#get-api)
* [`GET /api/cameras/<uuid>/`](#get-apicamerasuuid)
* [`GET /api/cameras/<uuid>/<stream>/recordings`](#get-apicamerasuuidstreamrecordings)
* [`GET /api/cameras/<uuid>/<stream>/view.mp4`](#get-apicamerasuuidstreamviewmp4)
* [`GET /api/cameras/<uuid>/<stream>/view.mp4.txt`](#get-apicamerasuuidstreamviewmp4txt)
* [`GET /api/cameras/<uuid>/<stream>/view.m4s`](#get-apicamerasuuidstreamviewm4s)
* [`GET /api/cameras/<uuid>/<stream>/view.m4s.txt`](#get-apicamerasuuidstreamviewm4stxt)
* [`GET /api/cameras/<uuid>/<stream>/live.m4s`](#get-apicamerasuuidstreamlivem4s)
* [`GET /api/init/<id>.mp4`](#get-apiinitidmp4)
* [`GET /api/init/<id>.mp4.txt`](#get-apiinitidmp4txt)
* [`GET /api/signals`](#get-apisignals)
* [`POST /api/signals`](#post-apisignals)
* [Request 1](#request-1)
* [Request 2](#request-2)
* [Request 3](#request-3)
## Objective
Allow a JavaScript-based web interface to list cameras and view recordings.
@ -704,7 +724,7 @@ Response:
}
```
### Request 3
#### Request 3
5 seconds later, the client observes motion has ended. It leaves the prior
data alone and predicts no more motion.

View File

@ -1,4 +1,4 @@
# Moonfire NVR Storage Schema
# Moonfire NVR Storage Schema <!-- omit in toc -->
Status: **current**.
@ -6,6 +6,20 @@ This is the initial design for the most fundamental parts of the Moonfire NVR
storage schema. See also [guide/schema.md](../guide/schema.md) for more
administrator-focused documentation.
* [Objective](#objective)
* [Cameras](#cameras)
* [Hard drives](#hard-drives)
* [Overview](#overview)
* [Detailed design](#detailed-design)
* [SQLite3](#sqlite3)
* [Duration of recordings](#duration-of-recordings)
* [Lifecycle of a sample file directory](#lifecycle-of-a-sample-file-directory)
* [Lifecycle of a recording](#lifecycle-of-a-recording)
* [Verifying invariants](#verifying-invariants)
* [Recording table](#recording-table)
* [`video_index`](#video_index)
* [<a href="on-demand"></a>On-demand `.mp4` construction](#on-demand-mp4-construction)
## Objective
Goals:
@ -73,13 +87,17 @@ different quality settings as well.
Decode:
```
$ time ffmpeg -y -threads 1 -i input.mp4 \
-f null /dev/null
```
Combo (Decode + encode with libx264):
```
$ time ffmpeg -y -threads 1 -i input.mp4 \
-c:v libx264 -preset ultrafast -threads 1 -f mp4 /dev/null
```
| Processor | 1080p30 decode | 1080p30 combo | 704x480p10 decode | 704x480p10 combo |
@ -115,8 +133,10 @@ only capable of 50 random accesses per second, and each one takes time that
otherwise could be used to transfer 2+ MB. The constrained resource, *disk
time fraction*, can be bounded as follows:
```
disk time fraction <= (seek rate) / (50 seeks/sec) +
(bandwidth) / (100 MB/sec)
```
## Overview
@ -131,15 +151,16 @@ Each recording is stored in two places:
Each file in this directory is simply a concatenation of the compressed,
timestamped video samples (also called "packets" or encoded frames), as
received from the camera. In MPEG-4 terminology (see [ISO
14496-12][iso-14496-12]), this is the contents of a `mdat` box for a `.mp4`
file representing the segment. These files do not contain framing data (start
and end byte offsets of samples) and thus are not meant to be decoded on
their own.
14496-12][iso-14496-12]), this is the contents of a `mdat` box for a
`.mp4` file representing the segment. These files do not contain framing
data (start and end byte offsets of samples) and thus are not meant to be
decoded on their own.
* the `recording` table in a [SQLite3][sqlite3] database, intended to be
stored on flash if possible. A row in this table contains all the metadata
associated with the segment, including the sample-by-sample contents of the
MPEG-4 `stbl` box. At 30 fps, a row is expected to require roughly 4 KB of
storage (2 bytes per sample, plus some fixed overhead).
stored on flash if possible. A row in this table contains all the
metadata associated with the segment, including the sample-by-sample
contents of the MPEG-4 `stbl` box. At 30 fps, a row is expected to
require roughly 4 KB of storage (2 bytes per sample, plus some fixed
overhead).
Putting the metadata on flash means metadata operations can be fast
(sub-millisecond random access, with parallelism) and do not take precious
@ -168,16 +189,17 @@ the SQLite3 database using [write-ahead logging][sqlite3-wal]. There are
several reasons for this decision:
* No user administration required. SQLite3, unlike its heavier-weight friends
MySQL and PostgreSQL, can be completely internal to the application. In many
applications, end users are unaware of the existence of a RDBMS, and
MySQL and PostgreSQL, can be completely internal to the application. In
many applications, end users are unaware of the existence of a RDBMS, and
Moonfire NVR should be no exception.
* Correctness. It's relatively easy to make guarantees about the state of an
ACID database, and SQLite3 in particular has a robust implementation. (See
[Files Are Hard][file-consistency].)
ACID database, and SQLite3 in particular has a robust implementation.
(See [Files Are Hard][file-consistency].)
* Developer ease and familiarity. SQL-based RDBMSs are quite common and
provide a lot of high-level constructs that ease development. SQLite3 in
particular is ubiquitous. Contributors are likely to come with some
understanding of the database, and there are many resources to learn more.
understanding of the database, and there are many resources to learn
more.
Total database size is expected to be roughly 4 KB per minute at 30 fps, or
1 GB for six camera-months of video. This will easily fit on a modest flash
@ -190,21 +212,23 @@ There are many constraints that influenced the choice of 1 minute as the
duration of recordings.
* Per-recording metadata size. There is a fixed component to the size of each
row, including the starting/ending timestamps, sample file UUID, etc. This
should not cause the database to be too large to fit on low-cost flash
devices. As described in the previous section, with 1 minute recordings the
size is quite modest.
row, including the starting/ending timestamps, sample file UUID, etc.
This should not cause the database to be too large to fit on low-cost
flash devices. As described in the previous section, with 1 minute
recordings the size is quite modest.
* Disk seeks. Sample files should be large enough that even during
simultaneous recording and playback of several streams, the disk seeks
incurred when switching from one file to another should not be significant.
At the extreme, a sample file per frame could cause an unacceptable 240
seeks per second just to record 8 30 fps streams. At one minute recording
time, 16 recording streams (2 per each of 8 cameras) and 4 playback streams
would cause on average 20 seeks per minute, or under 1% disk time.
incurred when switching from one file to another should not be
significant. At the extreme, a sample file per frame could cause an
unacceptable 240 seeks per second just to record 8 30 fps streams. At one
minute recording time, 16 recording streams (2 per each of 8 cameras) and
4 playback streams would cause on average 20 seeks per minute, or under
1% disk time.
* Internal fragmentation. Common Linux filesystems have a block size of 4 KiB
(see `statvfs.f_frsize`). Up to this much space per file will be wasted at
the end of each file. At the bitrates described in "Background", this is an
insignicant .02% waste for main streams and .5% waste for sub streams.
(see `statvfs.f_frsize`). Up to this much space per file will be wasted
at the end of each file. At the bitrates described in "Background", this
is an insignicant .02% waste for main streams and .5% waste for sub
streams.
* Number of "slices" in .mp4 files. As described [below](#on-demand),
`.mp4` files will be constructed on-demand for export. It should be
possible to export an hours-long segment without too much overhead. In
@ -214,8 +238,8 @@ duration of recordings.
* Crashes. On program crash or power loss, ideally it's acceptable to simply
discard any recordings in progress rather than add a checkpointing scheme.
* Granularity of retention. It should be possible to extend retention time
around motion events without forcing retention of too much additional data
or copying bytes around on disk.
around motion events without forcing retention of too much additional
data or copying bytes around on disk.
The design avoids the need for the following constraints:
@ -234,13 +258,14 @@ inconsistency. There are many ways this could arise:
failure or misconfiguration.
* the administrator mixing up the mount points of two filesystems holding
different sample file directories.
* the administrator renaming a sample file directory without updating the database.
* the administrator renaming a sample file directory without updating the
database.
* the administrator restoring the database from backup but not the sample file
directory, or vice versa.
* the administrator providing two sample file directory paths pointed at the
same inode via symlinks or non-canonical paths. (Note that flock(2) has a
design flaw in which multiple file descriptors can share a lock, so the current
locking scheme is not sufficient to detect this otherwise.)
design flaw in which multiple file descriptors can share a lock, so the
current locking scheme is not sufficient to detect this otherwise.)
* database and sample file directories forked from the same version, opened
the same number of times, then crossed.
@ -372,11 +397,11 @@ Precondition: database open read-write.
2. Verify the metadata file matches the database (as above).
3. Update the metadata file with `in_progress_open` matching the current
open.
3. Update the database row with `last_complete_open_id` matching the current
4. Update the database row with `last_complete_open_id` matching the current
open.
4. Update the metadata file with `last_complete_open` rather than
5. Update the metadata file with `last_complete_open` rather than
`in_progress_open`.
5. Run the recording startup procedure for this directory.
6. Run the recording startup procedure for this directory.
*Close a sample file directory*
@ -487,6 +512,7 @@ The times are roughly:
The `readdir()` and `fstat()` times can be tested simply:
```
$ mkdir testdir
$ cd testdir
$ seq 1 $[60*24*365*6/12*2] | xargs touch
@ -494,6 +520,7 @@ The `readdir()` and `fstat()` times can be tested simply:
$ time ls -1 -f | wc -l
$ sudo sh -c 'echo 1 > /proc/sys/vm/drop_caches'
$ time ls -1 -f --size | wc -l
```
(The system calls used by `ls` can be verified through strace.)
@ -515,6 +542,7 @@ the background at low priority.
The snippet below is a illustrative excerpt of the SQLite schema; see
`schema.sql` for the authoritative, up-to-date version.
```sql
-- A single, typically 60-second, recorded segment of video.
create table recording (
id integer primary key,
@ -551,6 +579,7 @@ The snippet below is a illustrative excerpt of the SQLite schema; see
-- type (avcC in the case of H.264).
data blob
);
```
As mentioned by the `start_time_90k` field above, recordings use a 90 kHz time
base. This matches the RTP timestamp frequency used for H.264 and other video
@ -614,6 +643,7 @@ This encoding is chosen so that values will be near zero, and thus the varints
will be at their most compact possible form. An index might be written by the
following pseudocode:
```
prev_duration = 0
prev_bytes_key = 0
prev_bytes_nonkey = 0
@ -624,6 +654,7 @@ following pseudocode:
if key: prev_bytes_key = bytes else: prev_bytes_nonkey = bytes
PutVarint((Zigzag(duration_delta) << 1) | is_key)
PutVarint(Zigzag(bytes_delta)
```
See also the example below:
@ -654,38 +685,6 @@ from several segments. The implementation details are outside the scope of this
document, but this is possible in part due to the use of an on-flash database
to store metadata and the simple, consistent format of sample indexes.
### Copyright
This file is part of Moonfire NVR, a security camera network video recorder.
Copyright (C) 2016 The Moonfire NVR Authors
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
In addition, as a special exception, the copyright holders give
permission to link the code of portions of this program with the
OpenSSL library under certain conditions as described in each
individual source file, and distribute linked combinations including
the two.
You must obey the GNU General Public License in all respects for all
of the code used other than OpenSSL. If you modify file(s) with this
exception, you may extend this exception to your version of the
file(s), but you are not obligated to do so. If you do not wish to do
so, delete this exception statement from your version. If you delete
this exception statement from all source files in the program, then
also delete it here.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
[pi2]: https://www.raspberrypi.org/products/raspberry-pi-2-model-b/
[xandem]: http://www.xandemhome.com/
[hikcam]: http://overseas.hikvision.com/us/Products_accessries_10533_i7696.html

View File

@ -1,4 +1,4 @@
# Moonfire NVR Time Handling
# Moonfire NVR Time Handling <!-- omit in toc -->
Status: **current**.
@ -7,6 +7,19 @@ Status: **current**.
>
> — Segal's law
* [Objective](#objective)
* [Background](#background)
* [Overview](#overview)
* [Detailed design](#detailed-design)
* [Caveats](#caveats)
* [Stream mismatches](#stream-mismatches)
* [Time discontinuities](#time-discontinuities)
* [Leap seconds](#leap-seconds)
* [Use `clock_gettime(CLOCK_TAI, ...)` timestamps](#use-clock_gettimeclock_tai--timestamps)
* [Use a leap second table when calculating differences](#use-a-leap-second-table-when-calculating-differences)
* [Use smeared time](#use-smeared-time)
* [Alternatives considered](#alternatives-considered)
## Objective
Maximize the likelihood Moonfire NVR's timestamps are useful.

View File

@ -1,4 +1,4 @@
# Building Moonfire NVR
# Building Moonfire NVR <!-- omit in toc -->
This document has notes for software developers on building Moonfire NVR from
source code for development. If you just want to install precompiled
@ -10,7 +10,6 @@ tracker](https://github.com/scottlamb/moonfire-nvr/issues) or
[mailing list](https://groups.google.com/d/forum/moonfire-nvr-users) when
stuck. Please also send pull requests to improve this doc.
* [Building Moonfire NVR](#building-moonfire-nvr)
* [Downloading](#downloading)
* [Docker builds](#docker-builds)
* [Release procedure](#release-procedure)
@ -152,8 +151,8 @@ Linux VM and filesystem overlay.
To build the server, you will need the following C libraries installed:
* [ffmpeg](http://ffmpeg.org/) version 2.x or 3.x, including `libavutil`,
`libavcodec` (to inspect H.264 frames), and `libavformat` (to connect to RTSP
servers and write `.mp4` files).
`libavcodec` (to inspect H.264 frames), and `libavformat` (to connect to
RTSP servers and write `.mp4` files).
Note ffmpeg library versions older than 55.1.101, along with all versions of
the competing project [libav](http://libav.org), don't support socket

View File

@ -1,4 +1,8 @@
# Working on UI development
# Working on UI development <!-- omit in toc -->
* [Getting started](#getting-started)
* [Overriding defaults](#overriding-defaults)
* [A note on `https`](#a-note-on-https)
The UI is presented from a single HTML page (index.html) and any number
of Javascript files, css files, images, etc. These are "packed" together

View File

@ -1,4 +1,11 @@
# Downloading, installing, and configuring Moonfire NVR with Docker
# Installing Moonfire NVR <!-- omit in toc -->
* [Downloading, installing, and configuring Moonfire NVR with Docker](#downloading-installing-and-configuring-moonfire-nvr-with-docker)
* [Dedicated hard drive setup](#dedicated-hard-drive-setup)
* [Completing configuration through the UI](#completing-configuration-through-the-ui)
* [Starting it up](#starting-it-up)
## Downloading, installing, and configuring Moonfire NVR with Docker
This document describes how to download, install, and configure Moonfire NVR
via the prebuilt Docker images available for x86-64, arm64, and arm. If you
@ -102,7 +109,7 @@ $ nvr init
This will create a directory `/var/lib/moonfire-nvr/db` with a SQLite3 database
within it.
## Dedicated hard drive setup
### Dedicated hard drive setup
If a dedicated hard drive is available, set up the mount point:
@ -139,7 +146,7 @@ mount lines. It will look similar to this:
--mount=type=bind,source=/media/nvr/sample,destination=/media/nvr/sample
```
## Completing configuration through the UI
### Completing configuration through the UI
Once your system is set up, it's time to initialize an empty database
and add the cameras and sample directories. You can do this
@ -171,16 +178,16 @@ In the user interface,
* `flush_if_sec` should typically be 120 seconds. This causes the database to
be flushed when the first instant of one of this stream's completed
recordings is 2 minutes old. A "recording" is a segment of a video
stream that is 60120 seconds when first establishing the stream, about
60 seconds midstream, and shorter when an error or server shutdown
terminates the stream. Thus, a value just below 60 will cause the
database to be flushed once per minute per stream in the steady state. A
value around 180 will cause the database to be once every 3 minutes per
stream, or less frequently if other streams cause flushes first. Lower
values cause less video to be lost on power loss. Higher values reduce
wear on the SSD holding the SQLite database, particularly when you have
many cameras and when you record both the "main" and "sub" streams of
each camera.
stream that is 60120 seconds when first establishing the stream,
about 60 seconds midstream, and shorter when an error or server
shutdown terminates the stream. Thus, a value just below 60 will
cause the database to be flushed once per minute per stream in the
steady state. A value around 180 will cause the database to be once
every 3 minutes per stream, or less frequently if other streams cause
flushes first. Lower values cause less video to be lost on power
loss. Higher values reduce wear on the SSD holding the SQLite
database, particularly when you have many cameras and when you record
both the "main" and "sub" streams of each camera.
3. Assign disk space to your cameras back in "Directories and retention".
Leave a little slack between the total limit and the filesystem capacity,
@ -202,7 +209,7 @@ In the user interface,
4. Add a user for yourself (and optionally others) under "Users". You'll need
this to access the web UI once you enable authentication.
## Starting it up
### Starting it up
Note that at this stage, Moonfire NVR's web interface is **insecure**: it
doesn't use `https` and doesn't require you to authenticate

View File

@ -1,4 +1,12 @@
# Moonfire NVR Schema Guide
# Moonfire NVR Schema Guide <!-- omit in toc -->
* [Upgrading](#upgrading)
* [Procedure](#procedure)
* [Unversioned to version 0](#unversioned-to-version-0)
* [Version 0 to version 1](#version-0-to-version-1)
* [Version 1 to version 2 to version 3](#version-1-to-version-2-to-version-3)
* [Version 3 to version 4 to version 5](#version-3-to-version-4-to-version-5)
* [Version 6](#version-6)
This document has notes about the Moonfire NVR storage schema. As described in
[README.md](../README.md), this consists of two kinds of state:
@ -56,12 +64,16 @@ Next ensure Moonfire NVR is not running and does not automatically restart if
the system is rebooted during the upgrade. If you followed the Docker
instructions, you can do this as follows:
```
$ nvr stop
```
Then back up your SQLite database. If you are using the default path, you can
do so as follows:
```
$ sudo -u moonfire-nvr cp /var/lib/moonfire-nvr/db/db{,.pre-upgrade}
```
By default, the upgrade command will reset the SQLite `journal_mode` to
`delete` prior to the upgrade. This works around a problem with
@ -117,7 +129,7 @@ for restore; neither are easy:
incorrectly believe they are still present. You could wait until all
existing files are rotated away, or you could try to delete them
manually from the database.
* if the new system created any recordings, the old system will not
* If the new system created any recordings, the old system will not
know about them and will not delete them. Your disk may become full.
You should find some way to discover these files and manually delete
them.
@ -150,8 +162,10 @@ Then use `sqlite3` to manually edit the database. The default
path is `/var/lib/moonfire-nvr/db/db`; if you've specified a different
`--db_dir`, use that directory with a suffix of `/db`.
```
$ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db
sqlite3>
```
At the prompt, run the following commands:

View File

@ -1,4 +1,16 @@
# Securing Moonfire NVR and exposing it to the Internet
# Securing Moonfire NVR and exposing it to the Internet <!-- omit in toc -->
* [The problem](#the-problem)
* [VPN or port forwarding?](#vpn-or-port-forwarding)
* [Overview](#overview)
* [1. Install a webserver](#1-install-a-webserver)
* [2. Configure a static internal IP](#2-configure-a-static-internal-ip)
* [3. Set up port forwarding](#3-set-up-port-forwarding)
* [4. Configure a public DNS name](#4-configure-a-public-dns-name)
* [5. Install a TLS certificate](#5-install-a-tls-certificate)
* [6. Reconfigure Moonfire NVR](#6-reconfigure-moonfire-nvr)
* [7. Configure the webserver](#7-configure-the-webserver)
* [Verify it works](#verify-it-works)
## The problem

View File

@ -1,10 +1,9 @@
# Troubleshooting
# Troubleshooting <!-- omit in toc -->
Here are some tips for diagnosing various problems with Moonfire NVR. Feel free
to open an [issue](https://github.com/scottlamb/moonfire-nvr/issues) if you
need more help.
* [Troubleshooting](#troubleshooting)
* [Viewing Moonfire NVR's logs](#viewing-moonfire-nvrs-logs)
* [Flushes](#flushes)
* [Panic errors](#panic-errors)