MyFavMirrors
/
moonfire-nvr
mirror of https://github.com/scottlamb/moonfire-nvr.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
moonfire-nvr/guide/schema.md

266 lines
11 KiB
Markdown
Raw Normal View History

version the sqlite3 database schema See guide/schema.md for instructions on upgrading past this commit.
2016-12-20 18:44:04 -05:00
# Moonfire NVR Schema Guide
This document has notes about the Moonfire NVR storage schema. As described in
[README.md](../README.md), this consists of two kinds of state:
* a SQLite database, typically <1 GiB. It should be stored on flash if
available.
* the "sample file directory", which holds the actual samples/frames of
H.264 video. This should be quite large and typically is stored on a hard
drive.
## Upgrading
The database schema includes a version number to quickly identify if a
the database is compatible with a particular version of the software. Some
software upgrades will require you to upgrade the database.
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
Note that in general upgrades are one-way and backward-incompatible. That is,
you can't downgrade the database to the old version, and you can't run the old
software on the new database. To minimize the corresponding risk, you should
save a backup of the old SQLite database and verify the new software works in
read-only mode prior to deleting the old database.
### Procedure
small improvements to schema upgrade instructions
2017-01-02 01:58:27 -05:00
First ensure there is sufficient space available for four copies of the
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
SQLite database:
more fixes to the upgrade guide
2018-01-31 00:14:53 -05:00
* copy 1: the copy to upgrade
* copy 2: a backup you manually create so that you can restore if you
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
discover a problem while running the new software against the upgraded
database in read-only mode. If disk space is tight, you can save this
to a different filesystem than the primary copy.
more fixes to the upgrade guide
2018-01-31 00:14:53 -05:00
* copies 3 and 4: internal copies made and destroyed by Moonfire NVR and
SQLite during the upgrade:
small improvements to schema upgrade instructions
2017-01-02 01:58:27 -05:00
* during earlier steps, possibly duplicate copies of tables, which
may occupy space both in the main database and the journal
* during the final vacuum step, a complete database copy
more fixes to the upgrade guide
2018-01-31 00:14:53 -05:00
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
If disk space is tight, and you are _very careful_, you can skip these
copies with the `--preset-journal=off --no-vacuum` arguments to
the updater. If you aren't confident in your ability to do this, *don't
do it*. If you are confident, take additional safety precautions anyway:
more fixes to the upgrade guide
2018-01-31 00:14:53 -05:00
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
* double-check you have the full backup described above. Without the
journal any problems during the upgrade will corrupt your database
and you will need to restore.
* ensure you re-enable journalling via `pragma journal_mode = wal;`
before using the upgraded database, or any problems after the
upgrade will corrupt your database. The upgrade procedure should do
this automatically, but you will want to verify by hand that you are
no longer in the dangerous mode.
Next ensure Moonfire NVR is not running and does not automatically restart if
update docs to recommend new Docker setup and remove the old scripts, to reduce the supported ways of doing things.
2021-01-21 19:00:38 -05:00
the system is rebooted during the upgrade. If you followed the Docker
instructions, you can do this as follows:
version the sqlite3 database schema See guide/schema.md for instructions on upgrading past this commit.
2016-12-20 18:44:04 -05:00
update docs to recommend new Docker setup and remove the old scripts, to reduce the supported ways of doing things.
2021-01-21 19:00:38 -05:00
$ nvr stop
version the sqlite3 database schema See guide/schema.md for instructions on upgrading past this commit.
2016-12-20 18:44:04 -05:00
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
Then back up your SQLite database. If you are using the default path, you can
do so as follows:
fix several formatting problems in upgrade guide
2018-01-30 20:01:03 -05:00
$ sudo -u moonfire-nvr cp /var/lib/moonfire-nvr/db/db{,.pre-upgrade}
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
By default, the upgrade command will reset the SQLite `journal_mode` to
`delete` prior to the upgrade. This works around a problem with
`journal_mode = wal` in older SQLite versions, as documented in [the SQLite
manual for write-ahead logging](https://www.sqlite.org/wal.html):
> WAL works best with smaller transactions. WAL does not work well for very
> large transactions. For transactions larger than about 100 megabytes,
> traditional rollback journal modes will likely be faster. For transactions
> in excess of a gigabyte, WAL mode may fail with an I/O or disk-full error.
> It is recommended that one of the rollback journal modes be used for
> transactions larger than a few dozen megabytes. Beginning with version
> 3.11.0 (2016-02-15), WAL mode works as efficiently with large transactions
> as does rollback mode.
update docs to recommend new Docker setup and remove the old scripts, to reduce the supported ways of doing things.
2021-01-21 19:00:38 -05:00
Run the upgrade procedure using the new software binary.
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
update docs to recommend new Docker setup and remove the old scripts, to reduce the supported ways of doing things.
2021-01-21 19:00:38 -05:00
```
$ nvr pull # updates the docker image to the latest binary
$ nvr upgrade # runs the upgrade
```
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
address database upgrade slowness (#107) * give a rule of thumb for update time in the documentation * log the SQLite3 version, which can affect performance * do the vacuum in non-WAL mode, to correctly set the page size and to avoid very slow behavior on older SQLite3 versions. Larger page sizes are generally faster (including subsequent vacuum operations). This won't help much for the first vacuum after this change, but it will help afterward. * likewise, set the page size properly on "moonfire-nvr init".
2021-02-11 00:44:06 -05:00
As a rule of thumb, on a Raspberry Pi 4 with a 1 GiB database, an upgrade might
take about four minutes for each schema version and for the final vacuum.
Next, you can run the system in read-only mode, although you'll find this only
update docs to recommend new Docker setup and remove the old scripts, to reduce the supported ways of doing things.
2021-01-21 19:00:38 -05:00
works in the "insecure" setup. (Authorization requires writing the database.)
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
update docs to recommend new Docker setup and remove the old scripts, to reduce the supported ways of doing things.
2021-01-21 19:00:38 -05:00
```
$ nvr rm
$ nvr run --read-only
```
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
Go to the web interface and ensure the system is operating correctly. If
update docs to recommend new Docker setup and remove the old scripts, to reduce the supported ways of doing things.
2021-01-21 19:00:38 -05:00
you detect a problem now, you can copy the old database back over the new one
and edit your `nvr` script to use the corresponding older Docker image. If
you detect a problem after enabling read-write operation, a restore will be
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
more complicated.
update docs to recommend new Docker setup and remove the old scripts, to reduce the supported ways of doing things.
2021-01-21 19:00:38 -05:00
Once you're satisfied, restart the system in read-write mode:
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
update docs to recommend new Docker setup and remove the old scripts, to reduce the supported ways of doing things.
2021-01-21 19:00:38 -05:00
```
$ nvr stop
$ nvr rm
$ nvr run
```
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
Hopefully your system is functioning correctly. If not, there are two options
for restore; neither are easy:
* go back to your old database. There will be two classes of problems:
* If the new system deleted any recordings, the old system will
incorrectly believe they are still present. You could wait until all
existing files are rotated away, or you could try to delete them
manually from the database.
* if the new system created any recordings, the old system will not
know about them and will not delete them. Your disk may become full.
You should find some way to discover these files and manually delete
them.
fix several formatting problems in upgrade guide
2018-01-30 20:01:03 -05:00
* undo the changes by hand. There's no documentation on this; you'll need
to read the code and come up with a reverse transformation.
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
update docs to recommend new Docker setup and remove the old scripts, to reduce the supported ways of doing things.
2021-01-21 19:00:38 -05:00
The `nvr check` command will show you what problems exist on your system.
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
### Unversioned to version 0
Early versions of Moonfire NVR (prior to 2016-12-20) did not include the
version information in the schema. You can manually add this information to
your schema using the `sqlite3` commandline. This process is backward
compatible, meaning that software versions that accept an unversioned database
will also accept a version 0 database.
Version 0 makes two changes:
fix several formatting problems in upgrade guide
2018-01-30 20:01:03 -05:00
* it adds schema versioning, as described above.
* it adds a column (`video_sync_samples`) to a database index to speed up
certain operations.
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
There's a special procedure for this upgrade. The good news is that a backup
is unnecessary; there's no risk with this procedure.
First ensure Moonfire NVR is not running as described in the general procedure
above.
Then use `sqlite3` to manually edit the database. The default
path is `/var/lib/moonfire-nvr/db/db`; if you've specified a different
`--db_dir`, use that directory with a suffix of `/db`.
version the sqlite3 database schema See guide/schema.md for instructions on upgrading past this commit.
2016-12-20 18:44:04 -05:00
fix several formatting problems in upgrade guide
2018-01-30 20:01:03 -05:00
$ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db
sqlite3>
version the sqlite3 database schema See guide/schema.md for instructions on upgrading past this commit.
2016-12-20 18:44:04 -05:00
At the prompt, run the following commands:
```sql
begin transaction;
create table version (
id integer primary key,
unix_time integer not null,
notes text
);
insert into version values (0, cast(strftime('%s', 'now') as int),
'manual upgrade to version 0');
drop index recording_cover;
create index recording_cover on recording (
camera_id,
start_time_90k,
duration_90k,
video_samples,
video_sample_entry_id,
sample_file_bytes
);
commit transaction;
```
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
When you are done, you can restart the service via `systemctl` and continue
using it with your existing or new version of Moonfire NVR.
### Version 0 to version 1
Version 1 makes several changes to the recording tables and indices. These
changes allow overlapping recordings to be unambiguously listed and viewed.
They also reduce the amount of I/O; in one test of retrieving playback
indexes, the number of (mostly 1024-byte) read syscalls on the database
dropped from 605 to 39.
version the sqlite3 database schema See guide/schema.md for instructions on upgrading past this commit.
2016-12-20 18:44:04 -05:00
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
The general upgrade procedure applies to this upgrade.
upgrader for v1->v2
2018-01-31 00:09:13 -05:00
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 13:11:10 -05:00
### Version 1 to version 2 to version 3
upgrader for v1->v2
2018-01-31 00:09:13 -05:00
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 13:11:10 -05:00
This upgrade affects the sample file directory as well as the database. Thus,
schema version 6 with pixel aspect ratio This makes anamorphic sub streams display correctly, even ones from old Hikvision cameras that don't properly set the aspect ratio at the H.264 layer.
2020-03-20 00:35:42 -04:00
the restore procedure written above of simply copying back the database is
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 13:11:10 -05:00
insufficient. To do a full restore, you would need to back up and restore the
sample file directory as well. This directory is considerably larger, so
consider an alternate procedure of crossing your fingers, and being prepared
to start over from scratch if there's a problem.
Version 2 represents a half-finished upgrade from version 1 to version 3; it
is never used.
Version 3 adds over version 1:
store rfc6381_codec in the database This avoids having codec-specific logic to synthesize it in db.rs. It's not too much of a problem now with only H.264 support, but it'd be a pain when supporting H.265 and other codecs.
2018-02-05 14:57:59 -05:00
initial schema for user authentication (#26) This is only the database schema, which I'm adding now in the hopes of freezing schema version 3. There's no way yet to create users, much less actually authenticate.
2018-03-22 02:55:21 -04:00
* user authentication
store rfc6381_codec in the database This avoids having codec-specific logic to synthesize it in db.rs. It's not too much of a problem now with only H.264 support, but it'd be a pain when supporting H.265 and other codecs.
2018-02-05 14:57:59 -05:00
* recording of sub streams (splits a new `stream` table out of `camera`)
knob to reduce db commits (SSD write cycles) This improves the practicality of having many streams (including the doubling of streams by having main + sub streams for each camera). With these tuned properly, extra streams don't cause any extra write cycles in normal or error cases. Consider the worst case in which each RTSP session immediately sends a single frame and then fails. Moonfire retries every second, so this would formerly cause one commit per second per stream. (flush_if_sec=0 preserves this behavior.) Now the commits can be arbitrarily infrequent by setting higher values of flush_if_sec. WARNING: this isn't production-ready! I hacked up dir.rs to make tests pass and "moonfire-nvr run" work in the best-case scenario, but it doesn't handle errors gracefully. I've been debating what to do when writing a recording fails. I considered "abandoning" the recording then either reusing or skipping its id. (in the latter case, marking the file as garbage if it can't be unlinked immediately). I think now there's no point in abandoning a recording. If I can't write to that file, there's no reason to believe another will work better. It's better to retry that recording forever, and perhaps put the whole directory into an error state that stops recording until those writes go through. I'm planning to redesign dir.rs to make this happen.
2018-02-22 19:35:34 -05:00
* a per-stream knob `flush_if_sec` meant to reduce database commits (and
thus SSD write cycles). This improves practicality of many streams.
support multiple sample file directories This is still pretty basic support. There's no config UI support for renaming/moving the sample file directories after they are created, and no error checking that the files are still in the expected place. I can imagine sysadmins getting into trouble trying to change things. I hope to address at least some of that in a follow-up change to introduce a versioning/locking scheme that ensures databases and sample file dirs match in some way. A bonus change that kinda got pulled along for the ride: a dialog pops up in the config UI while a stream is being tested. The experience was pretty bad before; there was no indication the button worked at all until it was done, sometimes many seconds later.
2018-02-12 01:45:51 -05:00
* support for multiple sample file directories, to take advantage of
multiple hard drives (or multiple RAID volumes).
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 13:11:10 -05:00
* an interlock between database and sample file directories to avoid various
new database/sample file dir interlock scheme The idea is to avoid the problems described in src/schema.proto; those possibilities have bothered me for a while. A bonus is that (in a future commit) it can replace the sample file uuid scheme in favor of using <camera_uuid>-<stream_type>/<recording_id> for several advantages: * on data integrity problems (specifically, extra sample files), more information to use to understand what happened. * no more reserving sample files prior to using them. This avoids some extra database transactions on startup (now there's an extra two total rather than an extra one per stream). It also simplifies an upcoming change I want to make in which some streams are not flushed immediately, reducing the write load significantly (maybe one per minute total rather than one per stream per minute). * get rid of eight bytes per playback cache entry in RAM (and nine bytes per recording_playback row on flash). The implementation is still pretty rough in places: * Lack of tests. * Poor ode organization. In particular, SampleFileDirectory::write_meta shouldn't be exposed beyond db. I'm thinking about moving db.rs and SampleFileDirectory to a new crate, moonfire_nvr_db. This would improve compile times as well. * No tooling for renaming a sample file directory. * Config subcommand still panics in conditions that can be reasonably expected to happen.
2018-02-15 02:10:10 -05:00
mixups that could cause data integrity problems.
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 13:11:10 -05:00
* recording the RFC-6381 codec associated with a video sample entry, so that
store rfc6381_codec in the database This avoids having codec-specific logic to synthesize it in db.rs. It's not too much of a problem now with only H.264 support, but it'd be a pain when supporting H.265 and other codecs.
2018-02-05 14:57:59 -05:00
logic for determining this is no longer needed as part of the database
layer.
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 13:11:10 -05:00
* a simpler sample file directory layout in which files are represented by
the same sequentially increasing id as in the database, rather than a
separate uuid which has to be reserved in advance.
support additional recording_integrity timestamps These are not actually populated by the code yet. I'm trying to get the v3 schema frozen as soon as possible; actually using the fields can come later. Add some explanation of their value in time.md, along with some general musing on leap seconds, and a correction on the frequency error of my cameras.
2018-03-22 01:32:41 -04:00
* additional timestamp fields which may be useful in diagnosing/correcting
time jumps/inconsistencies.
tweak and document version 3->4 upgrade
2019-06-20 19:03:59 -04:00
Use fixed-size directory meta files Add a new schema version 5; now 4 means the directory meta may or may not be upgraded. Fixes #65: now it's possible to open the directory even if it lies on a completely full disk.
2019-07-05 00:22:45 -04:00
### Version 3 to version 4 to version 5
tweak and document version 3->4 upgrade
2019-06-20 19:03:59 -04:00
schema version 6 with pixel aspect ratio This makes anamorphic sub streams display correctly, even ones from old Hikvision cameras that don't properly set the aspect ratio at the H.264 layer.
2020-03-20 00:35:42 -04:00
This upgrade affects the SQLite database and the sample file directory's
`meta` files.
Use fixed-size directory meta files Add a new schema version 5; now 4 means the directory meta may or may not be upgraded. Fixes #65: now it's possible to open the directory even if it lies on a completely full disk.
2019-07-05 00:22:45 -04:00
schema version 6 with pixel aspect ratio This makes anamorphic sub streams display correctly, even ones from old Hikvision cameras that don't properly set the aspect ratio at the H.264 layer.
2020-03-20 00:35:42 -04:00
Version 4 represents a half-finished upgrade from version 3 to version 5.
Use fixed-size directory meta files Add a new schema version 5; now 4 means the directory meta may or may not be upgraded. Fixes #65: now it's possible to open the directory even if it lies on a completely full disk.
2019-07-05 00:22:45 -04:00
Version 5 adds over version 3:
tweak and document version 3->4 upgrade
2019-06-20 19:03:59 -04:00
* permissions for users and sessions. Existing users will have only the
`view_video` permission, matching their previous behavior.
* the `signals` schema, used to store status of signals such as camera
note caveat on signals schema
2019-12-28 09:00:38 -05:00
motion detection, security system zones, etc. Note that while the schema
is stable for now, there's no support yet for configuring signals via
the `moonfire-nvr config` subcommand.
Use fixed-size directory meta files Add a new schema version 5; now 4 means the directory meta may or may not be upgraded. Fixes #65: now it's possible to open the directory even if it lies on a completely full disk.
2019-07-05 00:22:45 -04:00
* the ability to recover from a completely full sample file directory (#65)
without manual intervention.
schema version 6 with pixel aspect ratio This makes anamorphic sub streams display correctly, even ones from old Hikvision cameras that don't properly set the aspect ratio at the H.264 layer.
2020-03-20 00:35:42 -04:00
prepare to merge schema version 6 to master
2020-12-22 22:53:29 -05:00
### Version 6
schema version 6 with pixel aspect ratio This makes anamorphic sub streams display correctly, even ones from old Hikvision cameras that don't properly set the aspect ratio at the H.264 layer.
2020-03-20 00:35:42 -04:00
This upgrade affects only the SQLite database.
Version 6 adds over version 5:
* metadata about the pixel aspect ratio to properly support
[anamorphic](https://en.wikipedia.org/wiki/Anamorphic_widescreen) "sub"
streams.
use Blake3 instead of SHA-1 or Blake2b Benefits: * Blake3 is faster. This is most noticeable for the hashing of the sample file data. * we no longer need OpenSSL, which helps with shrinking the binary size (#70). sha1 basically forced OpenSSL usage; ring deliberately doesn't support this old algorithm, and the pure-Rust sha1 crate is painfully slow. OpenSSL might still be a better choice than ring/rustls for TLS but it's nice to have the option. For the video sample entries, I decided we don't need to hash at all. I think the id number is sufficiently stable, and it's okay---perhaps even desirable---if an existing init segment changes for fixes like e5b83c2.
2020-03-20 23:52:30 -04:00
* hashes in Blake3 rather than older SHA-1 (for file integrity checksums)
or Blake2b (for sessions).
track cumulative duration and runs This is useful for a combo scrub bar-based UI (#32) + live view UI (#59) in a non-obvious way. When constructing a HTML Media Source Extensions API SourceBuffer, the caller can specify a "mode" of either "segments" or "sequence": In "sequence" mode, playback assumes segments are added sequentially. This is good enough for a live view-only UI (#59) but not for a scrub bar UI in which you may want to seek backward to a segment you've never seen before. You will then need to insert a segment out-of-sequence. Imagine what happens when the user goes forward again until the end of the segment inserted immediately before it. The user should see the chronologically next segment or a pause for loading if it's unavailable. The best approximation of this is to track the mapping of timestamps to segments and insert a VTTCue with an enter/exit handler that seeks to the right position. But seeking isn't instantaneous; the user will likely briefly see first the segment they seeked to before. That's janky. Additionally, the "canplaythrough" event will behave strangely. In "segments" mode, playback respects the timestamps we set: * The obvious choice is to use wall clock timestamps. This is fine if they're known to be fixed and correct. They're not. The currently-recording segment may be "unanchored", meaning its start timestamp is not yet fixed. Older timestamps may overlap if the system clock was stepped between runs. The latter isn't /too/ bad from a user perspective, though it's confusing as a developer. We probably will only end up showing the more recent recording for a given timestamp anyway. But the former is quite annoying. It means we have to throw away part of the SourceBuffer that we may want to seek back (causing UI pauses when that happens) or keep our own spare copy of it (memory bloat). I'd like to avoid the whole mess. * Another approach is to use timestamps that are guaranteed to be in the correct order but that may have gaps. In particular, a timestamp of (recording_id * max_recording_duration) + time_within_recording. But again seeking isn't instantaneous. In my experiments, there's a visible pause between segments that drives me nuts. * Finally, the approach that led me to this schema change. Use timestamps that place each segment after the one before, possibly with an intentional gap between runs (to force a wait where we have an actual gap). This should make the browser's natural playback behavior work properly: it never goes to an incorrect place, and it only waits when/if we want it to. We have to maintain a mapping between its timestamps and segment ids but that's doable. This commit is only the schema change; the new data aren't exposed in the API yet, much less used by a UI. Note that stream.next_recording_id became stream.cum_recordings. I made a slight definition change in the process: recording ids for new streams start at 0 rather than 1. Various tests changed accordingly. The upgrade process makes a best effort to backfill these new fields, but of course it doesn't know the total duration or number of runs of previously deleted rows. That's good enough.
2020-06-09 19:17:32 -04:00
* for each recording row, the cumulative total duration and "runs" recorded
before it on that stream. This is useful for MediaSourceExtension-based
web browser UIs when setting timestamps of video segments in the
SourceBuffer.
start splitting wall and media duration for #34 This splits the schema and playback path. The recording path still adjusts the frame durations and always says the wall and media durations are the same. I expect to change that in a following commit. I wouldn't be surprised if that shakes out some bugs in this portion.
2020-08-05 00:44:01 -04:00
* decoupled "wall time" and "media time" of recoridngs, as a step toward
implementing audio support without giving up clock frequency adjustment. See
[this comment](https://github.com/scottlamb/moonfire-nvr/issues/34#issuecomment-651548468).
use Blake3 instead of SHA-1 or Blake2b Benefits: * Blake3 is faster. This is most noticeable for the hashing of the sample file data. * we no longer need OpenSSL, which helps with shrinking the binary size (#70). sha1 basically forced OpenSSL usage; ring deliberately doesn't support this old algorithm, and the pure-Rust sha1 crate is painfully slow. OpenSSL might still be a better choice than ring/rustls for TLS but it's nice to have the option. For the video sample entries, I decided we don't need to hash at all. I think the id number is sufficiently stable, and it's okay---perhaps even desirable---if an existing init segment changes for fixes like e5b83c2.
2020-03-20 23:52:30 -04:00
add a preliminary schema for object detection (#30)
2020-03-21 18:43:51 -04:00
On upgrading to this version, sessions will be revoked.