Merge branch 'master' (early part) into new-schema

Catch the new-schema branch up with everything up to (but not including) the
big UI refactoring. I'll merge that separately.
This commit is contained in:
Scott Lamb
2018-03-24 22:29:40 -07:00
27 changed files with 6187 additions and 827 deletions

146
guide/developing-ui.md Normal file
View File

@@ -0,0 +1,146 @@
# Working on UI development
The UI is presented from a single HTML page (index.html) and any number
of Javascript files, css files, images, etc. These are "packed" together
using [webpack](https://webpack.js.org).
For ongoing development it is possible to have the UI running in a web
browser using "hot loading". This means that as you make changes to source
files, they will be detected, the webpack will be recompiled and generated
and then the browser will be informed to reload things. In combination with
the debugger built into modern browsers this makes for a reasonable process.
For a production build, the same process is followed, except with different
settings. In particular, no hot loading development server will be started
and more effort is expended on packing and minimizing the components of
the application as represented in the various "bundles". Read more about
this in the webpack documentation.
## Getting started
Checkout the branch you want to work on and type
$ yarn start
This will pack and prepare a development setup. By default the development
server that serves up the web page(s) will listen on
[http://localhost:3000](http://localhost:3000) so you can direct your browser
there.
Make any changes to the source code as you desire (look at existing code
for examples and typical style), and the browser will hot-load your changes.
Often times you will make mistakes. Anything from a coding error (for which
you can use the browser's debugger), or compilation breaking Javascript errors.
The latter will often be reported as errors during the webpack assembly
process, but some will show up in the browser console, or both.
## Control and location of settings
Much of the settings needed to put the UI together, run webpack etc. is
located in a series of files that contain webpack configuration. These
files all live in the "webpack" subdirectory. We will not explain all
of them here, as you should rarely need to understand them, let alone
modify them.
What is worth mentioning is that the `package.json` file is configured
to use a different webpack configuration for development vs. production
builds. Both configurations depend on a shared configuration common to
both.
There are also some settings that control aspects of the MoonFire UI
behavior, such as window titles etc. These settings are found in the
`settings-nvr.js` file in the project root directory. They should be
pretty self explanatory.
The webpack configuration for all webpack builds is able to load the
values from `settings-nvr.js` and then, if the file exists, load those
from `settings-nvr-local.js` and use them to add to the configuration,
or replace. You can take advantage of this to add your own configuration
values in the "local" file, which does not get checked in, but is used
to affect development, and production builds.
## Special considerations for API calls
The UI code will make calls to the MoonFire NVR's server API, which is
assumed to run on the same host as the MoonFire server. This makes sense
because that server is also normally the one serving up the UI. For UI
development, however this is not always convenient, or even useful.
For one, you may be doing this development on a machine other than where
the main MoonFire server is running. That can work, but of course that
machine will not be responding the the API calls. If the UI does not
gracefully handle API failure errors (it should but development on that
is ongoing), it may break your UI code.
Therefore, for practical purposes, you may want the API calls to go to
a different server than where the localhost is. Rather than editing the
`webpack/dev.conf.js` file to make this happen, you should use a different
mechanism. The reason for not modifying this file (unless the change is
needed by all), is that the file is under source control and thus should
not reflect settings that are just for your personal use.
The manner in which you can achieve using a different server for API
calls is by telling the development server to use a "proxy" for
certain urls. All API calls start with "/api", so we'll take advantage
of that. Create a the file `settings-nvr-local.js` right next to the standard
`settings-nvr.js`. The file should look something like this:
module.exports.settings = {
devServer: {
proxy: {
'/api': 'http://192.168.1.232:8080'
}
}
};
This tells the development server to proxy all urls that it encounters
requests for to the url specified. The way this is done is by taking the
full URI component for the original request, and appending it to the string
configured on the right side of the path to be rewritten above. In a standard
MoonFire install, the server (and thus the API server as well), will be
listening on port 8080 a the IP address where the server lives. So adjust
the above for reach your own MoonFire instance where there is a server
running with some real data behind it.
### Issues with "older" MoonFire builds
You might also have though to change the "/api" string in the source code
to include the IP address of the MoonFire server. This would use the
"right" (desirable) URLs, but the requests will fail due to a violation
of the Cross-Origin Resource Sharing (CORS) protocol. If you really
need to, you can add a configuration option to the MoonFire server
by modifying its "service definition". We will not explain here how.
## Changing development server configuration
You can find our standard configuration for the development server inside
the `webpacks/dev.conf.js` file. Using the technique outlined above you
can change ports, ip addresses etc. One example where this may come in
useful is that you may want to "test" your new API code, running on
machine "A" (from a development server), proxying API requests to machine
"B" (because it has real data), from a browser running on machine "C".
The development server normally is configured to listing on port 3000
on "localhost." (which would correspond to machine "A" in this example).
However, you cannot use "localhost" on another machine to refer to "A".
You may think that using the IP address of "A" works, but it doesn't
because "localhost" lives at an IP address local to machine "A".
To make this work you must tell the development server on host "A" to
listen differently. You need to configure it to listen on IP address
"0.0.0.0", which means "all available interfaces". Once that is in
place you can use the IP address to reach "A" from "C". "A" will then
send API requests on to "B", and present final UI using information
from "A" and "B" to the browser on "C".
Modify the local settings to something like this:
module.exports.settings = {
devServer: {
host: "0.0.0.0",
proxy: {
'/api': 'http://192.168.1.232:8080'
}
}
};

145
guide/easy-install.md Normal file
View File

@@ -0,0 +1,145 @@
# Installing Moonfire NVR using provided scripts
There are no binary packages of Moonfire NVR available yet, so it must be built
from source. This is made easy using a few scripts that will do the job for you
unless you have very a different operating system. The scripts are written and
tested under ubuntu and raspbian but should not be hard to modify if necessary.
You'll start by downloading Moonfire if you have not already done so.
## Downloading
See the [github page](https://github.com/scottlamb/moonfire-nvr) (in case
you're not reading this text there already). You can download the bleeding
edge version from the command line via git:
$ git clone https://github.com/scottlamb/moonfire-nvr.git
## Preparation steps for easy install
There are a few things to prepare if you want a truly turnkey install, but
they are both optional.
### Dedicated media directory
An optional, but strongly suggested, step is to setup a dedicated hard disk
for recording video.
Moonfire works best if the video samples are collected on a hard drive of
sufficient capacity and separate from the root and main file systems. This
is particularly important on Raspberry Pi based systems as the flash based
main file systems have a limited lifetime and are way too small to hold
any significant amount of video.
If a dedicated hard drive is available, set up the mount point (in this
example we'll use /media/nvr/samples):
$ sudo vim /etc/fstab
$ sudo mount /mount/media/samples
In the fstab you would add a line similar to this:
/dev/disk/by-uuid/23d550bc-0e38-4825-acac-1cac8a7e091f /media/nvr ext4 defaults,noatime 0 1
You'll have to lookup the correct uuid for your disk. One way to do that is
to issue the following commands:
$ ls -l /dev/disk/by-uuid
Locate the device where your disk will be mounted (or is mounted), for example
`/dev/sda1`. Now lookup the filename linked to that from the output of the
`ls` command. This is the uuid you need.
The setup script (see below) will create the necessary sample file dir on the mounted
hard disk.
## Setting everything up
Start by executing the setup script:
$ cd moonfire-nvr
$ scripts/setup-ubuntu.sh
If this is the very first time you run this script, a file named `prep.config`
will be created and the script will stop. This file is where you will set
or change variables that describe the Moonfire installation you want. The
initial execution will put default values in this value, but only for the
most commonly changed variables. For a full list of variables, see below.
Once you modify this file (if desired), you run the setup script again. This
time it will use the values in the file and proceed with the setup.
The script will download and install any pre-requisites. Watch carefully for
error messages. It may be you have conflicting installations. If that is the
case you must either resolve those first, or go the manual route.
The script may be given the "-f" option. If you do, you are telling the script
that you do not want any existing installation of ffmpeg to be overwritten with
a newer one. This could be important to you. If you do use it, and the version
you have installed is not compatible with Moonfire, you will be told through
a message. If you have no ffmpeg installed, the option is effectively ignored
and the necessary version of ffmpeg will be installed.
The setup script should only need to be run once (after `prep.config` has been
created), although if you do a re-install of Moonfire, in particular a much
newer version, it is a good idea to run it again as requirements and pre-requisites
may have changed. Running the script multiple times should not have any negative effects.
*WARNING* It is quite possible that during the running of the setup script,
in particular during the building of libavutil you will see several compiler
warnings. This, while undesirable, is a direct result of the original
developers not cleaning up the cause(s) of these warnings. They are, however,
just warnings and will not affect correct functioning of Moonfire.
Once the setup is complete, two steps remain: building and then installing.
There is a script for each of these scenarios, but since generally you would
want to install after a succesul build, the build script automatically invokes
the install script, unless specifically told not to.
## Building
The build script is involved like this:
$ scripts/build.sh
This script will perform all steps necessary to build a complete Moonfire
setup. If there are no build errors, this script will then automatically
invoke the install script (see below).
There are two options you may pass to this script. The first is "-B" which
means "build only". In other words, this will stop the automatic invocation
of the install script. The other option available is "-t" and causes the
script to ignore the results of any tests. In other words, even if tests
fail, the build phase will be considered successful. This can occasionally
be useful if you are doing development, and have temporarily broken one
or more test, but want to proceed anyway.
## Installing
The install step is performed by the script that can be manually invoked
like this:
$ scripts/install.sh
This script will copy various files resulting from the build to the correct
locations. It will also create a "service configuration" for systemctl that
can be used to control Moonfire. This service configuration can be prevented
by using the "-s" option to this script. It will also prevent the automatic
start of this configuration.
## Configuration variables
Although not all listed in the default prep.config file, these are the
available configuration variable and their defaults. In the most frequent
scenarios you will probably only change SAMPLE_MEDIA_DIR to point
to your mounted external disk (/media/nvr in the example above).
NVR_USER=moonfire-nvr
NVR_GROUP=$NVR_USER
NVR_PORT=8080
NVR_HOME_BASE=/var/lib
DB_NAME=db
DB_DIR=$NVR_HOME/$DB_NAME
SAMPLE_FILE_DIR=sample
SAMPLE_MEDIA_DIR=$NVR_HOME
SERVICE_NAME=moonfire-nvr
SERVICE_DESC="Moonfire NVR"
SERVICE_BIN=/usr/local/bin/$SERVICE_NAME

201
guide/install-manual.md Normal file
View File

@@ -0,0 +1,201 @@
# Installing Moonfire NVR
This document describes how to install Moonfire NVR on a Linux system.
## Downloading
See the [github page](https://github.com/scottlamb/moonfire-nvr) (in case
you're not reading this text there already). You can download the bleeding
edge version from the command line via git:
$ git clone https://github.com/scottlamb/moonfire-nvr.git
## Building from source
There are no binary packages of Moonfire NVR available yet, so it must be built
from source.
Moonfire NVR is written in the [Rust Programming
Language](https://www.rust-lang.org/en-US/). In the long term, I expect this
will result in a more secure, full-featured, easy-to-install software.
You will need the following C libraries installed:
* [ffmpeg](http://ffmpeg.org/) version 2.x or 3.x, including `libavutil`,
`libavcodec` (to inspect H.264 frames), and `libavformat` (to connect to RTSP
servers and write `.mp4` files).
Note ffmpeg library versions older than 55.1.101, along with all versions of
the competing project [libav](http://libav.org), don't support socket
timeouts for RTSP. For reliable reconnections on error, it's strongly
recommended to use ffmpeg library versions >= 55.1.101.
* [SQLite3](https://www.sqlite.org/).
* [`ncursesw`](https://www.gnu.org/software/ncurses/), the UTF-8 version of
the `ncurses` library.
On recent Ubuntu or Raspbian, the following command will install
all non-Rust dependencies:
$ sudo apt-get install \
build-essential \
libavcodec-dev \
libavformat-dev \
libavutil-dev \
libncurses5-dev \
libncursesw5-dev \
libsqlite3-dev \
libssl-dev \
pkgconf
Next, you need Rust 1.17+ and Cargo. The easiest way to install them is by following
the instructions at [rustup.rs](https://www.rustup.rs/).
Finally, building the UI requires [yarn](https://yarnpkg.com/en/).
You can continue to follow the build/install instructions below for a manual
build and install, or alternatively you can run the prep script called `prep.sh`.
$ cd moonfire-nvr
$ ./prep.sh
The script will take the following command line options, should you need them:
* `-S`: Skip updating and installing dependencies through apt-get. This too can be
useful on repeated builds.
You can edit variables at the start of the script to influence names and
directories, but defaults should suffice in most cases. For details refer to
the script itself. We will mention just one option, needed when you follow the
suggestion to separate database and samples between flash storage and a hard disk.
If you have the hard disk mounted on, lets say `/media/nvr`, and you want to
store the video samples inside a directory named `samples` there, you would set:
SAMPLE_FILE_DIR=/media/nvr/samples
The script will perform all necessary steps to leave you with a fully built,
installed moonfire-nvr binary. The only thing
you'll have to do manually is add your camera configuration(s) to the database.
Alternatively, before running the script, you can create a file named `cameras.sql`
in the same directory as the `prep.sh` script and it will be automatically
included for you.
For instructions, you can skip to "[Camera configuration and hard disk mounting](#camera)".
Once prerequisites are installed, Moonfire NVR can be built as follows:
$ yarn
$ yarn build
$ cargo test
$ cargo build --release
$ sudo install -m 755 target/release/moonfire-nvr /usr/local/bin
$ sudo mkdir /usr/local/lib/moonfire-nvr
$ sudo cp -R ui-dist /usr/local/lib/moonfire-nvr/ui
## Further configuration
Moonfire NVR should be run under a dedicated user. It keeps two kinds of
state:
* a SQLite database, typically <1 GiB. It should be stored on flash if
available.
* the "sample file directory", which holds the actual samples/frames of
H.264 video. This should be quite large and typically is stored on a hard
drive.
(See [schema.md](schema.md) for more information.)
Both kinds of state are intended to be accessed only by Moonfire NVR itself.
However, the interface for adding new cameras is not yet written, so you will
have to manually insert cameras with the `sqlite3` command line tool prior to
starting Moonfire NVR.
Manual commands would look something like this:
$ sudo addgroup --system moonfire-nvr
$ sudo adduser --system moonfire-nvr --home /var/lib/moonfire-nvr
$ sudo mkdir /var/lib/moonfire-nvr
$ sudo -u moonfire-nvr -H mkdir db sample
$ sudo -u moonfire-nvr moonfire-nvr init
### <a name="cameras"></a>Camera configuration and hard drive mounting
If a dedicated hard drive is available, set up the mount point:
$ sudo vim /etc/fstab
$ sudo mount /var/lib/moonfire-nvr/sample
Once setup is complete, it is time to add camera configurations to the
database. If the daemon is running, you will need to stop it temporarily:
$ sudo systemctl stop moonfire-nvr
You can configure the system through a text-based user interface:
$ sudo -u moonfire-nvr moonfire-nvr config 2>debug-log
In the user interface, add your cameras under the "Edit cameras" dialog.
There's a "Test" button to verify your settings directly from the dialog.
After the cameras look correct, go to "Edit retention" to assign disk space to
each camera. Leave a little slack (at least 100 MB per camera) between the total
limit and the filesystem capacity, even if you store nothing else on the disk.
There are several reasons this is needed:
* The limit currently controls fully-written files only. There will be up
to two minutes of video per camera of additional video.
* The rotation happens after the limit is exceeded, not proactively.
* Moonfire NVR currently doesn't account for the unused space in the final
filesystem block at the end of each file.
* Moonfire NVR doesn't account for the space used for directory listings.
* If a file is open when it is deleted (such as if a HTTP client is
downloading it), it stays around until the file is closed. Moonfire NVR
currently doesn't account for this.
When finished, start the daemon:
$ sudo systemctl start moonfire-nvr
### System Service
Moonfire NVR can be run as a systemd service. If you used `prep.sh` this has
been done for you. If not, Create
`/etc/systemd/system/moonfire-nvr.service`:
[Unit]
Description=Moonfire NVR
After=network-online.target
[Service]
ExecStart=/usr/local/bin/moonfire-nvr run \
--db-dir=/var/lib/moonfire-nvr/db \
--http-addr=0.0.0.0:8080
Environment=TZ=:/etc/localtime
Environment=MOONFIRE_FORMAT=google-systemd
Environment=MOONFIRE_LOG=info
Environment=RUST_BACKTRACE=1
Type=simple
User=moonfire-nvr
Nice=-20
Restart=on-abnormal
CPUAccounting=true
MemoryAccounting=true
BlockIOAccounting=true
[Install]
WantedBy=multi-user.target
Note that the HTTP port currently has no authentication, encryption, or
logging; it should not be directly exposed to the Internet.
Complete the installation through `systemctl` commands:
$ sudo systemctl daemon-reload
$ sudo systemctl start moonfire-nvr
$ sudo systemctl status moonfire-nvr
$ sudo systemctl enable moonfire-nvr
See the [systemd](http://www.freedesktop.org/wiki/Software/systemd/)
documentation for more information. The [manual
pages](http://www.freedesktop.org/software/systemd/man/) for `systemd.service`
and `systemctl` may be of particular interest.

View File

@@ -13,84 +13,17 @@ edge version from the command line via git:
## Building from source
There are no binary packages of Moonfire NVR available yet, so it must be built
from source.
from source. To do so, you can follow two paths:
Moonfire NVR is written in the [Rust Programming
Language](https://www.rust-lang.org/en-US/). In the long term, I expect this
will result in a more secure, full-featured, easy-to-install software.
* Scripted install: You will run some shell scripts (after preparing one or
two files, and will be completely done. This is by far the easiest option,
in particular for first time builders/installers. This process is fully
described. Read more in [Easy Installation](easy-install.md)
* Manual build and install. This is explained in detail in these
[instructions](install-manual.md)
You will need the following C libraries installed:
* [ffmpeg](http://ffmpeg.org/) version 2.x or 3.x, including `libavutil`,
`libavcodec` (to inspect H.264 frames), and `libavformat` (to connect to RTSP
servers and write `.mp4` files).
Note ffmpeg library versions older than 55.1.101, along with all versions of
the competing project [libav](http://libav.org), don't not support socket
timeouts for RTSP. For reliable reconnections on error, it's strongly
recommended to use ffmpeg library versions >= 55.1.101.
* [SQLite3](https://www.sqlite.org/).
* [`ncursesw`](https://www.gnu.org/software/ncurses/), the UTF-8 version of
the `ncurses` library.
On recent Ubuntu or Raspbian, the following command will install
all non-Rust dependencies:
$ sudo apt-get install \
build-essential \
libavcodec-dev \
libavformat-dev \
libavutil-dev \
libncurses5-dev \
libncursesw5-dev \
libsqlite3-dev \
libssl-dev \
pkgconf
Next, you need Rust 1.17+ and Cargo. The easiest way to install them is by following
the instructions at [rustup.rs](https://www.rustup.rs/).
Finally, building the UI requires [yarn](https://yarnpkg.com/en/).
You can continue to follow the build/install instructions below for a manual
build and install, or alternatively you can run the prep script called `prep.sh`.
$ cd moonfire-nvr
$ ./prep.sh
The script will take the following command line options, should you need them:
* `-S`: Skip updating and installing dependencies through apt-get. This too can be
useful on repeated builds.
You can edit variables at the start of the script to influence names and
directories, but defaults should suffice in most cases. For details refer to
the script itself. We will mention just one option, needed when you follow the
suggestion to separate database and samples between flash storage and a hard disk.
If you have the hard disk mounted on, lets say `/media/nvr`, and you want to
store the video samples inside a directory named `samples` there, you would set:
SAMPLES_DIR=/media/nvr/samples
The script will perform all necessary steps to leave you with a fully built,
installed moonfire-nvr binary. The only thing
you'll have to do manually is add your camera configuration(s) to the database.
Alternatively, before running the script, you can create a file named `cameras.sql`
in the same directory as the `prep.sh` script and it will be automatically
included for you.
For instructions, you can skip to "[Camera configuration and hard disk mounting](#camera)".
Once prerequisites are installed, Moonfire NVR can be built as follows:
$ yarn
$ yarn build
$ cargo test
$ cargo build --release
$ sudo install -m 755 target/release/moonfire-nvr /usr/local/bin
$ sudo mkdir /usr/local/lib/moonfire-nvr
$ sudo cp -R ui-dist /usr/local/lib/moonfire-nvr/ui
Regardless of the approach for setup and installation above that you choose,
please read the further configuration instructions below.
## Further configuration
@@ -103,13 +36,13 @@ state:
H.264 video. These should be quite large and are typically stored on hard
drives.
Both states are intended to be accessed by moonfire-nvr only, but can be
changed after building. See below.
(See [schema.md](schema.md) for more information.)
Both kinds of state are intended to be accessed only by Moonfire NVR itself.
However, the interface for adding new cameras is not yet written, so you will
have to manually insert cameras with the `sqlite3` command line tool prior to
starting Moonfire NVR.
The database changes and sample file directory changes require the moonfire-nvr
binary to be built, so can only be done after completing the build. The other
settings and preparations should be done before building.
Manual commands would look something like this:
$ sudo addgroup --system moonfire-nvr
@@ -118,7 +51,7 @@ Manual commands would look something like this:
$ sudo -u moonfire-nvr -H mkdir db sample
$ sudo -u moonfire-nvr moonfire-nvr init
### <a name="cameras"></a>Camera configuration and hard drive mounting
### <a name="drive mounting"></a>Camera configuration and hard drive mounting
If a dedicated hard drive is available, set up the mount point:
@@ -130,7 +63,7 @@ database. If the daemon is running, you will need to stop it temporarily:
$ sudo systemctl stop moonfire-nvr
You can configure the system through a text-based user interface:
You can configure the system's database through a text-based user interface:
$ sudo -u moonfire-nvr moonfire-nvr config 2>debug-log