moonfire-nvr/src/schema.sql

143 lines
5.5 KiB
MySQL
Raw Normal View History

-- This file is part of Moonfire NVR, a security camera digital video recorder.
-- Copyright (C) 2016 Scott Lamb <slamb@slamb.org>
--
-- This program is free software: you can redistribute it and/or modify
-- it under the terms of the GNU General Public License as published by
-- the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- In addition, as a special exception, the copyright holders give
-- permission to link the code of portions of this program with the
-- OpenSSL library under certain conditions as described in each
-- individual source file, and distribute linked combinations including
-- the two.
--
-- You must obey the GNU General Public License in all respects for all
-- of the code used other than OpenSSL. If you modify file(s) with this
-- exception, you may extend this exception to your version of the
-- file(s), but you are not obligated to do so. If you do not wish to do
-- so, delete this exception statement from your version. If you delete
-- this exception statement from all source files in the program, then
-- also delete it here.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU General Public License for more details.
--
-- You should have received a copy of the GNU General Public License
-- along with this program. If not, see <http://www.gnu.org/licenses/>.
--
-- schema.sql: SQLite3 database schema for Moonfire NVR.
-- See also design/schema.md.
--pragma journal_mode = wal;
create table camera (
id integer primary key,
uuid blob unique,-- not null check (length(uuid) = 16),
-- A short name of the camera, used in log messages.
short_name text,-- not null,
-- A short description of the camera.
description text,
-- The host (or IP address) to use in rtsp:// URLs when accessing the camera.
host text,
-- The username to use when accessing the camera.
-- If empty, no username or password will be supplied.
username text,
-- The password to use when accessing the camera.
password text,
-- The path (starting with "/") to use in rtsp:// URLs to reference this
-- camera's "main" (full-quality) video stream.
main_rtsp_path text,
-- The path (starting with "/") to use in rtsp:// URLs to reference this
-- camera's "sub" (low-bandwidth) video stream.
sub_rtsp_path text,
-- The number of bytes of video to retain, excluding the currently-recording
-- file. Older files will be deleted as necessary to stay within this limit.
retain_bytes integer not null check (retain_bytes >= 0)
);
-- Each row represents a single completed recorded segment of video.
-- Recordings are typically ~60 seconds; never more than 5 minutes.
create table recording (
id integer primary key,
camera_id integer references camera (id) not null,
sample_file_bytes integer not null check (sample_file_bytes > 0),
-- The starting time of the recording, in 90 kHz units since
Write using the shiny new schema There's a lot of work left to do on this: * important latency optimization: the recording threads block while fsync()ing sample files, which can take 250+ ms. This should be moved to a separate thread to happen asynchronously. * write cycle optimizations: several SQLite commits per camera per minute. * test coverage: this drops testing of the file rotation, and there are several error paths worth testing. * ffmpeg oddities to investigate: * the out-of-order first frame's pts * measurable delay before returning packets * it sometimes returns an initial packet it calls a "key" frame that actually has an SEI recovery point NAL but not an IDR-coded slice NAL, even though in the input these always seem to come together. This makes playback starting from this recording not work at all on Chrome. The symptom is that it loads a player-looking thing with the proper dimensions but playback never actually starts. I imagine these are all related but haven't taken the time to dig through ffmpeg code and understand them. The right thing anyway may be to ditch ffmpeg for RTSP streaming (perhaps in favor of the live555 library), as it seems to have other omissions like making it hard/impossible to take advantage of Sender Reports. In the meantime, I attempted to mitigate problems by decreasing ffmpeg's probesize. * handling overlapping recordings: right now if there's too much time drift or a time jump, you can end up with recordings that the UI won't play without manual database changes. It's not obvious what the right thing to do is. * easy camera setup: currently you have to manually insert rows in the SQLite database and restart. but I think it's best to get something in to iterate from. This deletes a lot of code, including: * the ffmpeg video sink code (instead now using a bit of extra code in Stream on top of the SampleFileWriter, SampleIndexEncoder, and MoonfireDatabase code that's been around for a while) * FileManager (in favor of new code using the database) * the old UI * RealFile and friends * the dependency on protocol buffers, which was used for the config file (though I'll likely have other reasons for using protocol buffers later) * even some utilities like IsWord that were just for validating the config
2016-02-04 02:22:37 -05:00
-- 1970-01-01 00:00:00 UTC. Currently on initial connection, this is taken
-- from the local system time; on subsequent recordings, it exactly
-- matches the previous recording's end time.
start_time_90k integer not null check (start_time_90k > 0),
-- The duration of the recording, in 90 kHz units.
duration_90k integer not null
check (duration_90k >= 0 and duration_90k < 5*60*90000),
Write using the shiny new schema There's a lot of work left to do on this: * important latency optimization: the recording threads block while fsync()ing sample files, which can take 250+ ms. This should be moved to a separate thread to happen asynchronously. * write cycle optimizations: several SQLite commits per camera per minute. * test coverage: this drops testing of the file rotation, and there are several error paths worth testing. * ffmpeg oddities to investigate: * the out-of-order first frame's pts * measurable delay before returning packets * it sometimes returns an initial packet it calls a "key" frame that actually has an SEI recovery point NAL but not an IDR-coded slice NAL, even though in the input these always seem to come together. This makes playback starting from this recording not work at all on Chrome. The symptom is that it loads a player-looking thing with the proper dimensions but playback never actually starts. I imagine these are all related but haven't taken the time to dig through ffmpeg code and understand them. The right thing anyway may be to ditch ffmpeg for RTSP streaming (perhaps in favor of the live555 library), as it seems to have other omissions like making it hard/impossible to take advantage of Sender Reports. In the meantime, I attempted to mitigate problems by decreasing ffmpeg's probesize. * handling overlapping recordings: right now if there's too much time drift or a time jump, you can end up with recordings that the UI won't play without manual database changes. It's not obvious what the right thing to do is. * easy camera setup: currently you have to manually insert rows in the SQLite database and restart. but I think it's best to get something in to iterate from. This deletes a lot of code, including: * the ffmpeg video sink code (instead now using a bit of extra code in Stream on top of the SampleFileWriter, SampleIndexEncoder, and MoonfireDatabase code that's been around for a while) * FileManager (in favor of new code using the database) * the old UI * RealFile and friends * the dependency on protocol buffers, which was used for the config file (though I'll likely have other reasons for using protocol buffers later) * even some utilities like IsWord that were just for validating the config
2016-02-04 02:22:37 -05:00
-- The number of 90 kHz units the local system time is ahead of the
-- recording; negative numbers indicate the local system time is behind
-- the recording. Large values would indicate that the local time has jumped
-- during recording or that the local time and camera time frequencies do
-- not match.
local_time_delta_90k integer not null,
video_samples integer not null check (video_samples > 0),
video_sync_samples integer not null check (video_samples > 0),
video_sample_entry_id integer references video_sample_entry (id),
sample_file_uuid blob not null check (length(sample_file_uuid) = 16),
sample_file_sha1 blob not null check (length(sample_file_sha1) = 20),
video_index blob not null check (length(video_index) > 0)
);
create index recording_cover on recording (
-- Typical queries use "where camera_id = ? order by start_time_90k (desc)?".
camera_id,
start_time_90k,
-- These fields are not used for ordering; they cover most queries so
-- that only database verification and actual viewing of recordings need
-- to consult the underlying row.
duration_90k,
video_samples,
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
video_sync_samples,
video_sample_entry_id,
sample_file_bytes
);
-- Files in the sample file directory which may be present but should simply be
-- discarded on startup. (Recordings which were never completed or have been
-- marked for completion.)
create table reserved_sample_files (
uuid blob primary key check (length(uuid) = 16),
state integer not null -- 0 (writing) or 1 (deleted)
) without rowid;
-- A concrete box derived from a ISO/IEC 14496-12 section 8.5.2
-- VisualSampleEntry box. Describes the codec, width, height, etc.
create table video_sample_entry (
id integer primary key,
-- A SHA-1 hash of |bytes|.
sha1 blob unique not null check (length(sha1) = 20),
-- The width and height in pixels; must match values within
-- |sample_entry_bytes|.
width integer not null check (width > 0),
height integer not null check (height > 0),
-- The serialized box, including the leading length and box type (avcC in
-- the case of H.264).
data blob not null check (length(data) > 86)
);