MyFavMirrors/moonfire-nvr
1
0
Fork 0
mirror of https://github.com/scottlamb/moonfire-nvr.git synced 2025-01-03 19:13:23 -05:00
Code Issues Packages Projects Releases Wiki Activity
f3b17a4bd8
moonfire-nvr/src/mp4.rs

1807 lines
74 KiB
Rust
Raw Normal View History

Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
// This file is part of Moonfire NVR, a security camera digital video recorder.
// Copyright (C) 2016 Scott Lamb <slamb@slamb.org>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// In addition, as a special exception, the copyright holders give
// permission to link the code of portions of this program with the
// OpenSSL library under certain conditions as described in each
// individual source file, and distribute linked combinations including
// the two.
//
// You must obey the GNU General Public License in all respects for all
// of the code used other than OpenSSL. If you modify file(s) with this
// exception, you may extend this exception to your version of the
// file(s), but you are not obligated to do so. If you do not wish to do
// so, delete this exception statement from your version. If you delete
// this exception statement from all source files in the program, then
// also delete it here.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
//! `.mp4` virtual file serving.
//!
//! The `mp4` module builds virtual files representing ISO/IEC 14496-12 (ISO base media format /
//! MPEG-4 / `.mp4`) video. These can be constructed from one or more recordings and are suitable
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
//! for HTTP range serving or download. The generated `.mp4` file has the `moov` box before the
//! `mdat` box for fast start. More specifically, boxes are arranged in the order suggested by
//! ISO/IEC 14496-12 section 6.2.3 (Table 1):
//!
//! * ftyp (file type and compatibility)
//! * moov (container for all the metadata)
//! ** mvhd (movie header, overall declarations)
//!
//! ** trak (video: container for an individual track or stream)
//! *** tkhd (track header, overall information about the track)
//! *** (optional) edts (edit list container)
//! **** elst (an edit list)
//! *** mdia (container for the media information in a track)
//! **** mdhd (media header, overall information about the media)
//! *** minf (media information container)
//! **** vmhd (video media header, overall information (video track only))
//! **** dinf (data information box, container)
//! ***** dref (data reference box, declares source(s) of media data in track)
//! **** stbl (sample table box, container for the time/space map)
//! ***** stsd (sample descriptions (codec types, initilization etc.)
//! ***** stts ((decoding) time-to-sample)
//! ***** stsc (sample-to-chunk, partial data-offset information)
//! ***** stsz (samples sizes (framing))
//! ***** co64 (64-bit chunk offset)
//! ***** stss (sync sample table)
//!
//! ** (optional) trak (subtitle: container for an individual track or stream)
//! *** tkhd (track header, overall information about the track)
//! *** mdia (container for the media information in a track)
//! **** mdhd (media header, overall information about the media)
//! *** minf (media information container)
//! **** nmhd (null media header, overall information)
//! **** dinf (data information box, container)
//! ***** dref (data reference box, declares source(s) of media data in track)
//! **** stbl (sample table box, container for the time/space map)
//! ***** stsd (sample descriptions (codec types, initilization etc.)
//! ***** stts ((decoding) time-to-sample)
//! ***** stsc (sample-to-chunk, partial data-offset information)
//! ***** stsz (samples sizes (framing))
//! ***** co64 (64-bit chunk offset)
//!
//! * mdat (media data container)
//! ```
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
extern crate byteorder;
extern crate time;
use byteorder::{BigEndian, ByteOrder, WriteBytesExt};
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
use db;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
use dir;
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
use error::Error;
replace resource.rs with new http-entity crate This crate is a slightly-more-polished and MIT-licensed version of resource.rs. So far it has one advantage: running the tests doesn't require RUST_TEST_THREADS=1.
2016-12-20 21:29:45 -05:00
use http_entity;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
use hyper::header;
use mmapfile;
update several dependencies I left serde alone because uuid hasn't been updated for the new version.
2017-01-27 23:58:04 -05:00
use openssl::hash;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
use recording::{self, TIME_UNITS_PER_SEC};
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
use slices::{self, Slices};
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
use smallvec::SmallVec;
shrink mp4::Segment 128 -> 112 bytes (on 64-bit) * don't store sizes of mp4-format sample indexes; recalculate them. * keep SampleIndexIterator position as a u32 rather than a usize. This is 960 bytes for a 60-minute mp4; another small cache usage improvement.
2017-02-26 03:02:49 -05:00
use lazycell::LazyCell;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
use std::cmp;
use std::io;
use std::ops::Range;
use std::mem;
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
use std::sync::Arc;
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-09 00:28:50 -05:00
use strutil;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
use time::Timespec;
/// This value should be incremented any time a change is made to this file that causes different
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
/// bytes to be output for a particular set of `Builder` options. Incrementing this value will
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
/// cause the etag to change as well.
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
const FORMAT_VERSION: [u8; 1] = [0x03];
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
/// An `ftyp` (ISO/IEC 14496-12 section 4.3 `FileType`) box.
const FTYP_BOX: &'static [u8] = &[
0x00, 0x00, 0x00, 0x20, // length = 32, sizeof(FTYP_BOX)
b'f', b't', b'y', b'p', // type
b'i', b's', b'o', b'm', // major_brand
0x00, 0x00, 0x02, 0x00, // minor_version
b'i', b's', b'o', b'm', // compatible_brands[0]
b'i', b's', b'o', b'2', // compatible_brands[1]
b'a', b'v', b'c', b'1', // compatible_brands[2]
b'm', b'p', b'4', b'1', // compatible_brands[3]
];
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// An `hdlr` (ISO/IEC 14496-12 section 8.4.3 `HandlerBox`) box suitable for video.
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
const VIDEO_HDLR_BOX: &'static [u8] = &[
0x00, 0x00, 0x00, 0x21, // length == sizeof(kHdlrBox)
b'h', b'd', b'l', b'r', // type == hdlr, ISO/IEC 14496-12 section 8.4.3.
0x00, 0x00, 0x00, 0x00, // version + flags
0x00, 0x00, 0x00, 0x00, // pre_defined
b'v', b'i', b'd', b'e', // handler = vide
0x00, 0x00, 0x00, 0x00, // reserved[0]
0x00, 0x00, 0x00, 0x00, // reserved[1]
0x00, 0x00, 0x00, 0x00, // reserved[2]
0x00, // name, zero-terminated (empty)
];
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// An `hdlr` (ISO/IEC 14496-12 section 8.4.3 `HandlerBox`) box suitable for subtitles.
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
const SUBTITLE_HDLR_BOX: &'static [u8] = &[
0x00, 0x00, 0x00, 0x21, // length == sizeof(kHdlrBox)
b'h', b'd', b'l', b'r', // type == hdlr, ISO/IEC 14496-12 section 8.4.3.
0x00, 0x00, 0x00, 0x00, // version + flags
0x00, 0x00, 0x00, 0x00, // pre_defined
b's', b'b', b't', b'l', // handler = sbtl
0x00, 0x00, 0x00, 0x00, // reserved[0]
0x00, 0x00, 0x00, 0x00, // reserved[1]
0x00, 0x00, 0x00, 0x00, // reserved[2]
0x00, // name, zero-terminated (empty)
];
/// Part of an `mvhd` (`MovieHeaderBox` version 0, ISO/IEC 14496-12 section 8.2.2), used from
/// `append_mvhd`.
const MVHD_JUNK: &'static [u8] = &[
0x00, 0x01, 0x00, 0x00, // rate
0x01, 0x00, // volume
0x00, 0x00, // reserved
0x00, 0x00, 0x00, 0x00, // reserved
0x00, 0x00, 0x00, 0x00, // reserved
0x00, 0x01, 0x00, 0x00, // matrix[0]
0x00, 0x00, 0x00, 0x00, // matrix[1]
0x00, 0x00, 0x00, 0x00, // matrix[2]
0x00, 0x00, 0x00, 0x00, // matrix[3]
0x00, 0x01, 0x00, 0x00, // matrix[4]
0x00, 0x00, 0x00, 0x00, // matrix[5]
0x00, 0x00, 0x00, 0x00, // matrix[6]
0x00, 0x00, 0x00, 0x00, // matrix[7]
0x40, 0x00, 0x00, 0x00, // matrix[8]
0x00, 0x00, 0x00, 0x00, // pre_defined[0]
0x00, 0x00, 0x00, 0x00, // pre_defined[1]
0x00, 0x00, 0x00, 0x00, // pre_defined[2]
0x00, 0x00, 0x00, 0x00, // pre_defined[3]
0x00, 0x00, 0x00, 0x00, // pre_defined[4]
0x00, 0x00, 0x00, 0x00, // pre_defined[5]
];
/// Part of a `tkhd` (`TrackHeaderBox` version 0, ISO/IEC 14496-12 section 8.3.2), used from
/// `append_video_tkhd` and `append_subtitle_tkhd`.
const TKHD_JUNK: &'static [u8] = &[
0x00, 0x00, 0x00, 0x00, // reserved
0x00, 0x00, 0x00, 0x00, // reserved
0x00, 0x00, 0x00, 0x00, // layer + alternate_group
0x00, 0x00, 0x00, 0x00, // volume + reserved
0x00, 0x01, 0x00, 0x00, // matrix[0]
0x00, 0x00, 0x00, 0x00, // matrix[1]
0x00, 0x00, 0x00, 0x00, // matrix[2]
0x00, 0x00, 0x00, 0x00, // matrix[3]
0x00, 0x01, 0x00, 0x00, // matrix[4]
0x00, 0x00, 0x00, 0x00, // matrix[5]
0x00, 0x00, 0x00, 0x00, // matrix[6]
0x00, 0x00, 0x00, 0x00, // matrix[7]
0x40, 0x00, 0x00, 0x00, // matrix[8]
];
/// Part of a `minf` (`MediaInformationBox`, ISO/IEC 14496-12 section 8.4.4), used from
/// `append_video_minf`.
const VIDEO_MINF_JUNK: &'static [u8] = &[
b'm', b'i', b'n', b'f', // type = minf, ISO/IEC 14496-12 section 8.4.4.
// A vmhd box; the "graphicsmode" and "opcolor" values don't have any
// meaningful use.
0x00, 0x00, 0x00, 0x14, // length == sizeof(kVmhdBox)
b'v', b'm', b'h', b'd', // type = vmhd, ISO/IEC 14496-12 section 12.1.2.
0x00, 0x00, 0x00, 0x01, // version + flags(1)
0x00, 0x00, 0x00, 0x00, // graphicsmode (copy), opcolor[0]
0x00, 0x00, 0x00, 0x00, // opcolor[1], opcolor[2]
// A dinf box suitable for a "self-contained" .mp4 file (no URL/URN
// references to external data).
0x00, 0x00, 0x00, 0x24, // length == sizeof(kDinfBox)
b'd', b'i', b'n', b'f', // type = dinf, ISO/IEC 14496-12 section 8.7.1.
0x00, 0x00, 0x00, 0x1c, // length
b'd', b'r', b'e', b'f', // type = dref, ISO/IEC 14496-12 section 8.7.2.
0x00, 0x00, 0x00, 0x00, // version and flags
0x00, 0x00, 0x00, 0x01, // entry_count
0x00, 0x00, 0x00, 0x0c, // length
b'u', b'r', b'l', b' ', // type = url, ISO/IEC 14496-12 section 8.7.2.
0x00, 0x00, 0x00, 0x01, // version=0, flags=self-contained
];
/// Part of a `minf` (`MediaInformationBox`, ISO/IEC 14496-12 section 8.4.4), used from
/// `append_subtitle_minf`.
const SUBTITLE_MINF_JUNK: &'static [u8] = &[
b'm', b'i', b'n', b'f', // type = minf, ISO/IEC 14496-12 section 8.4.4.
// A nmhd box.
0x00, 0x00, 0x00, 0x0c, // length == sizeof(kNmhdBox)
b'n', b'm', b'h', b'd', // type = nmhd, ISO/IEC 14496-12 section 12.1.2.
0x00, 0x00, 0x00, 0x01, // version + flags(1)
// A dinf box suitable for a "self-contained" .mp4 file (no URL/URN
// references to external data).
0x00, 0x00, 0x00, 0x24, // length == sizeof(kDinfBox)
b'd', b'i', b'n', b'f', // type = dinf, ISO/IEC 14496-12 section 8.7.1.
0x00, 0x00, 0x00, 0x1c, // length
b'd', b'r', b'e', b'f', // type = dref, ISO/IEC 14496-12 section 8.7.2.
0x00, 0x00, 0x00, 0x00, // version and flags
0x00, 0x00, 0x00, 0x01, // entry_count
0x00, 0x00, 0x00, 0x0c, // length
b'u', b'r', b'l', b' ', // type = url, ISO/IEC 14496-12 section 8.7.2.
0x00, 0x00, 0x00, 0x01, // version=0, flags=self-contained
];
/// Part of a `stbl` (`SampleTableBox`, ISO/IEC 14496 section 8.5.1) used from
/// `append_subtitle_stbl`.
const SUBTITLE_STBL_JUNK: &'static [u8] = &[
b's', b't', b'b', b'l', // type = stbl, ISO/IEC 14496-12 section 8.5.1.
// A stsd box.
0x00, 0x00, 0x00, 0x54, // length
b's', b't', b's', b'd', // type == stsd, ISO/IEC 14496-12 section 8.5.2.
0x00, 0x00, 0x00, 0x00, // version + flags
0x00, 0x00, 0x00, 0x01, // entry_count == 1
// SampleEntry, ISO/IEC 14496-12 section 8.5.2.2.
0x00, 0x00, 0x00, 0x44, // length
b't', b'x', b'3', b'g', // type == tx3g, 3GPP TS 26.245 section 5.16.
0x00, 0x00, 0x00, 0x00, // reserved
0x00, 0x00, 0x00, 0x01, // reserved, data_reference_index == 1
// TextSampleEntry
0x00, 0x00, 0x00, 0x00, // displayFlags == none
0x00, // horizontal-justification == left
0x00, // vertical-justification == top
0x00, 0x00, 0x00, 0x00, // background-color-rgba == transparent
// TextSampleEntry.BoxRecord
0x00, 0x00, // top
0x00, 0x00, // left
0x00, 0x00, // bottom
0x00, 0x00, // right
// TextSampleEntry.StyleRecord
0x00, 0x00, // startChar
0x00, 0x00, // endChar
0x00, 0x01, // font-ID
0x00, // face-style-flags
0x12, // font-size == 18 px
0xff, 0xff, 0xff, 0xff, // text-color-rgba == opaque white
// TextSampleEntry.FontTableBox
0x00, 0x00, 0x00, 0x16, // length
b'f', b't', b'a', b'b', // type == ftab, section 5.16
0x00, 0x01, // entry-count == 1
0x00, 0x01, // font-ID == 1
0x09, // font-name-length == 9
b'M', b'o', b'n', b'o', b's', b'p', b'a', b'c', b'e',
];
/// Pointers to each static bytestrings.
/// The order here must match the `StaticBytestring` enum.
const STATIC_BYTESTRINGS: [&'static [u8]; 8] = [
FTYP_BOX,
VIDEO_HDLR_BOX,
SUBTITLE_HDLR_BOX,
MVHD_JUNK,
TKHD_JUNK,
VIDEO_MINF_JUNK,
SUBTITLE_MINF_JUNK,
SUBTITLE_STBL_JUNK,
];
/// Enumeration of the static bytestrings. The order here must match the `STATIC_BYTESTRINGS`
/// array. The advantage of this enum over direct pointers to the relevant strings is that it
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
/// fits into `Slice`'s 20-bit `p`.
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
#[derive(Copy, Clone, Debug)]
enum StaticBytestring {
FtypBox,
VideoHdlrBox,
SubtitleHdlrBox,
MvhdJunk,
TkhdJunk,
VideoMinfJunk,
SubtitleMinfJunk,
SubtitleStblJunk,
}
/// The template fed into strtime for a timestamp subtitle. This must produce fixed-length output
/// (see `SUBTITLE_LENGTH`) to allow quick calculation of the total size of the subtitles for
/// a given time range.
const SUBTITLE_TEMPLATE: &'static str = "%Y-%m-%d %H:%M:%S %z";
/// The length of the output of `SUBTITLE_TEMPLATE`.
const SUBTITLE_LENGTH: usize = 25; // "2015-07-02 17:10:00 -0700".len();
shrink mp4::Segment 128 -> 112 bytes (on 64-bit) * don't store sizes of mp4-format sample indexes; recalculate them. * keep SampleIndexIterator position as a u32 rather than a usize. This is 960 bytes for a 60-minute mp4; another small cache usage improvement.
2017-02-26 03:02:49 -05:00
/// The lengths of the indexes associated with a `Segment`; for use within `Segment` only.
struct SegmentLengths {
stts: usize,
stsz: usize,
stss: usize,
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// A wrapper around `recording::Segment` that keeps some additional `.mp4`-specific state.
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
struct Segment {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
s: recording::Segment,
shrink mp4::Segment 128 -> 112 bytes (on 64-bit) * don't store sizes of mp4-format sample indexes; recalculate them. * keep SampleIndexIterator position as a u32 rather than a usize. This is 960 bytes for a 60-minute mp4; another small cache usage improvement.
2017-02-26 03:02:49 -05:00
/// If generated, the `.mp4`-format sample indexes.
/// 1. stts: `slice[.. stsz_start]`
/// 2. stsz: `slice[stsz_start .. stss_start]`
/// 3. stss: `slice[stss_start ..]`
/// Access only through `write_index`.
index: LazyCell<Result<Box<[u8]>, ()>>,
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
/// The 1-indexed frame number in the `File` of the first frame in this segment.
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
first_frame_num: u32,
num_subtitle_samples: u32,
}
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
impl Segment {
shrink mp4::Segment 128 -> 112 bytes (on 64-bit) * don't store sizes of mp4-format sample indexes; recalculate them. * keep SampleIndexIterator position as a u32 rather than a usize. This is 960 bytes for a 60-minute mp4; another small cache usage improvement.
2017-02-26 03:02:49 -05:00
fn write_index<F>(&self, r: Range<u64>, db: &db::Database, out: &mut io::Write, f: F)
-> Result<(), Error>
where F: FnOnce(&[u8], SegmentLengths) -> &[u8] {
let lens = SegmentLengths {
stts: mem::size_of::<u32>() * 2 * (self.s.frames as usize),
stsz: mem::size_of::<u32>() * self.s.frames as usize,
stss: mem::size_of::<u32>() * self.s.key_frames as usize,
};
let index = self.index.borrow_with(|| {
self.build_index(&lens, db)
.map_err(|e| {
error!("Unable to build index for segment: {:?}", e);
()
})
});
let index = match *index {
Ok(ref b) => &b[..],
Err(()) => {
return Err(Error::new("Unable to build index; see previous error.".to_owned()))
},
};
out.write_all(&f(&index, lens)[r.start as usize .. r.end as usize])?;
Ok(())
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
shrink mp4::Segment 128 -> 112 bytes (on 64-bit) * don't store sizes of mp4-format sample indexes; recalculate them. * keep SampleIndexIterator position as a u32 rather than a usize. This is 960 bytes for a 60-minute mp4; another small cache usage improvement.
2017-02-26 03:02:49 -05:00
fn stts(buf: &[u8], lens: SegmentLengths) -> &[u8] { &buf[.. lens.stts] }
fn stsz(buf: &[u8], lens: SegmentLengths) -> &[u8] { &buf[lens.stts .. lens.stts + lens.stsz] }
fn stss(buf: &[u8], lens: SegmentLengths) -> &[u8] { &buf[lens.stts + lens.stsz ..] }
fn build_index(&self, lens: &SegmentLengths, db: &db::Database) -> Result<Box<[u8]>, Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let s = &self.s;
shrink mp4::Segment 128 -> 112 bytes (on 64-bit) * don't store sizes of mp4-format sample indexes; recalculate them. * keep SampleIndexIterator position as a u32 rather than a usize. This is 960 bytes for a 60-minute mp4; another small cache usage improvement.
2017-02-26 03:02:49 -05:00
let len = lens.stts + lens.stsz + lens.stss;
let mut buf = {
stop using a couple unstable features It would be nice to build on stable Rust. In particular, I'm hitting compiler bugs in Rust nightly, such at this one: https://github.com/rust-lang/rust/issues/38177 I imagine beta/stable compilers would be less problematic. These two features were easy to get rid of: * alloc was used to get a Box<[u8]> to uninitialized memory. Looks like that's possible with Vec. * box_syntax wasn't actually used at all. (Maybe a leftover from something.) The remaining features are: * plugin, for clippy. https://github.com/rust-lang/rust/issues/29597 I could easily gate it with a "nightly" cargo feature. * proc_macro, for serde_derive. https://github.com/rust-lang/rust/issues/35900 serde does support stable rust, although it's annoying. https://serde.rs/codegen-stable.html I might just wait a bit; this feature looks like it's getting close to stabilization.
2016-12-08 00:05:49 -05:00
let mut v = Vec::with_capacity(len);
shrink mp4::Segment 128 -> 112 bytes (on 64-bit) * don't store sizes of mp4-format sample indexes; recalculate them. * keep SampleIndexIterator position as a u32 rather than a usize. This is 960 bytes for a 60-minute mp4; another small cache usage improvement.
2017-02-26 03:02:49 -05:00
unsafe { v.set_len(len) };
stop using a couple unstable features It would be nice to build on stable Rust. In particular, I'm hitting compiler bugs in Rust nightly, such at this one: https://github.com/rust-lang/rust/issues/38177 I imagine beta/stable compilers would be less problematic. These two features were easy to get rid of: * alloc was used to get a Box<[u8]> to uninitialized memory. Looks like that's possible with Vec. * box_syntax wasn't actually used at all. (Maybe a leftover from something.) The remaining features are: * plugin, for clippy. https://github.com/rust-lang/rust/issues/29597 I could easily gate it with a "nightly" cargo feature. * proc_macro, for serde_derive. https://github.com/rust-lang/rust/issues/35900 serde does support stable rust, although it's annoying. https://serde.rs/codegen-stable.html I might just wait a bit; this feature looks like it's getting close to stabilization.
2016-12-08 00:05:49 -05:00
v.into_boxed_slice()
};
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
{
shrink mp4::Segment 128 -> 112 bytes (on 64-bit) * don't store sizes of mp4-format sample indexes; recalculate them. * keep SampleIndexIterator position as a u32 rather than a usize. This is 960 bytes for a 60-minute mp4; another small cache usage improvement.
2017-02-26 03:02:49 -05:00
let (stts, mut rest) = buf.split_at_mut(lens.stts);
let (stsz, stss) = rest.split_at_mut(lens.stsz);
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let mut frame = 0;
let mut key_frame = 0;
let mut last_start_and_dur = None;
s.foreach(db, |it| {
last_start_and_dur = Some((it.start_90k, it.duration_90k));
BigEndian::write_u32(&mut stts[8*frame .. 8*frame+4], 1);
BigEndian::write_u32(&mut stts[8*frame+4 .. 8*frame+8], it.duration_90k as u32);
BigEndian::write_u32(&mut stsz[4*frame .. 4*frame+4], it.bytes as u32);
if it.is_key {
BigEndian::write_u32(&mut stss[4*key_frame .. 4*key_frame+4],
self.first_frame_num + (frame as u32));
key_frame += 1;
}
frame += 1;
Ok(())
})?;
// Fix up the final frame's duration.
// Doing this after the fact is more efficient than having a condition on every
// iteration.
if let Some((last_start, dur)) = last_start_and_dur {
BigEndian::write_u32(&mut stts[8*frame-4 ..],
cmp::min(s.desired_range_90k.end - last_start, dur) as u32);
}
}
shrink mp4::Segment 128 -> 112 bytes (on 64-bit) * don't store sizes of mp4-format sample indexes; recalculate them. * keep SampleIndexIterator position as a u32 rather than a usize. This is 960 bytes for a 60-minute mp4; another small cache usage improvement.
2017-02-26 03:02:49 -05:00
Ok(buf)
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
}
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
pub struct FileBuilder {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
/// Segments of video: one per "recording" table entry as they should
/// appear in the video.
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
segments: Vec<Segment>,
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
video_sample_entries: SmallVec<[Arc<db::VideoSampleEntry>; 1]>,
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
next_frame_num: u32,
duration_90k: u32,
num_subtitle_samples: u32,
subtitle_co64_pos: Option<usize>,
body: BodyState,
include_timestamp_subtitle_track: bool,
}
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
/// The portion of `FileBuilder` which is mutated while building the body of the file.
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
/// This is separated out from the rest so that it can be borrowed in a loop over
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
/// `FileBuilder::segments`; otherwise this would cause a double-self-borrow.
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
struct BodyState {
use associated types for Slice This is more readable; in particular, it avoids the need for awkward the PhantomData in the Slices struct.
2017-02-25 21:54:52 -05:00
slices: Slices<Slice>,
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
/// `self.buf[unflushed_buf_pos .. self.buf.len()]` holds bytes that should be
/// appended to `slices` before any other slice. See `flush_buf()`.
unflushed_buf_pos: usize,
buf: Vec<u8>,
}
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
/// A single slice of a `File`, for use with a `Slices` object. Each slice is responsible for
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// some portion of the generated `.mp4` file. The box headers and such are generally in `Static`
/// or `Buf` slices; the others generally represent a single segment's contribution to the
/// like-named box.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
///
/// This is stored in a packed representation to be more cache-efficient:
///
/// * low 40 bits: end() (maximum 1 TiB).
/// * next 4 bits: t(), the SliceType.
/// * top 20 bits: p(), a parameter specified by the SliceType (maximum 1 Mi).
struct Slice(u64);
/// The type of a `Slice`.
#[derive(Copy, Clone, Debug)]
#[repr(u8)]
enum SliceType {
Static = 0, // param is index into STATIC_BYTESTRINGS
Buf = 1, // param is index into m.buf
VideoSampleEntry = 2, // param is index into m.video_sample_entries
Stts = 3, // param is index into m.segments
Stsz = 4, // param is index into m.segments
Co64 = 5, // param is unused
Stss = 6, // param is index into m.segments
VideoSampleData = 7, // param is index into m.segments
SubtitleSampleData = 8, // param is index into m.segments
// There must be no value > 15, as this is packed into 4 bits in Slice.
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
impl Slice {
fn new(end: u64, t: SliceType, p: usize) -> Result<Self, Error> {
if end >= (1<<40) || p >= (1<<20) {
return Err(Error::new(format!("end={} p={} too large for Slice", end, p)));
}
Ok(Slice(end | ((t as u64) << 40) | ((p as u64) << 44)))
}
}
impl Slice {
fn t(&self) -> SliceType {
// This value is guaranteed to be a valid SliceType because it was copied from a SliceType
// in Slice::new.
unsafe { ::std::mem::transmute(((self.0 >> 40) & 0xF) as u8) }
}
fn p(&self) -> usize { (self.0 >> 44) as usize }
}
use associated types for Slice This is more readable; in particular, it avoids the need for awkward the PhantomData in the Slices struct.
2017-02-25 21:54:52 -05:00
impl slices::Slice for Slice {
type Ctx = File;
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn end(&self) -> u64 { return self.0 & 0xFF_FF_FF_FF_FF }
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
fn write_to(&self, f: &File, r: Range<u64>, l: u64, out: &mut io::Write)
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
-> Result<(), Error> {
let t = self.t();
let p = self.p();
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
trace!("write {:?}, range {:?} out of len {}", self, r, l);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
match t {
SliceType::Static => {
let s = STATIC_BYTESTRINGS[p];
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let part = &s[r.start as usize .. r.end as usize];
out.write_all(part)?;
Ok(())
},
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
SliceType::Buf => {
out.write_all(&f.buf[p+r.start as usize .. p+r.end as usize])?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
Ok(())
},
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
SliceType::VideoSampleEntry => {
out.write_all(&f.video_sample_entries[p].data[r.start as usize .. r.end as usize])?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
Ok(())
},
shrink mp4::Segment 128 -> 112 bytes (on 64-bit) * don't store sizes of mp4-format sample indexes; recalculate them. * keep SampleIndexIterator position as a u32 rather than a usize. This is 960 bytes for a 60-minute mp4; another small cache usage improvement.
2017-02-26 03:02:49 -05:00
SliceType::Stts => f.segments[p].write_index(r, &f.db, out, Segment::stts),
SliceType::Stsz => f.segments[p].write_index(r, &f.db, out, Segment::stsz),
SliceType::Stss => f.segments[p].write_index(r, &f.db, out, Segment::stss),
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
SliceType::Co64 => f.write_co64(r, l, out),
SliceType::VideoSampleData => f.write_video_sample_data(p, r, out),
SliceType::SubtitleSampleData => f.write_subtitle_sample_data(p, r, l, out),
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
}
}
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
impl ::std::fmt::Debug for Slice {
fn fmt(&self, f: &mut ::std::fmt::Formatter) -> Result<(), ::std::fmt::Error> {
// Write an unpacked representation. Omit end(); Slices writes that part.
write!(f, "{:?} {}", self.t(), self.p())
}
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Converts from 90kHz units since Unix epoch (1970-01-01 00:00:00 UTC) to seconds since
/// ISO-14496 epoch (1904-01-01 00:00:00 UTC).
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
fn to_iso14496_timestamp(t: recording::Time) -> u32 { (t.unix_seconds() + 24107 * 86400) as u32 }
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Writes a box length for everything appended in the supplied scope.
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
/// Used only within FileBuilder::build (and methods it calls internally).
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
macro_rules! write_length {
($_self:ident, $b:block) => {{
let len_pos = $_self.body.buf.len();
let len_start = $_self.body.slices.len() + $_self.body.buf.len() as u64 -
$_self.body.unflushed_buf_pos as u64;
$_self.body.append_u32(0); // placeholder.
{ $b; }
let len_end = $_self.body.slices.len() + $_self.body.buf.len() as u64 -
$_self.body.unflushed_buf_pos as u64;
BigEndian::write_u32(&mut $_self.body.buf[len_pos .. len_pos + 4],
(len_end - len_start) as u32);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
Ok::<_, Error>(())
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}}
}
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
impl FileBuilder {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
pub fn new() -> Self {
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
FileBuilder{
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
segments: Vec::new(),
video_sample_entries: SmallVec::new(),
next_frame_num: 1,
duration_90k: 0,
num_subtitle_samples: 0,
subtitle_co64_pos: None,
body: BodyState{
slices: Slices::new(),
buf: Vec::new(),
unflushed_buf_pos: 0,
},
include_timestamp_subtitle_track: false,
}
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Sets if the generated `.mp4` should include a subtitle track with second-level timestamps.
/// Default is false.
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
pub fn include_timestamp_subtitle_track(&mut self, b: bool) {
self.include_timestamp_subtitle_track = b;
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Reserves space for the given number of additional segments.
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
pub fn reserve(&mut self, additional: usize) {
self.segments.reserve(additional);
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a segment for (a subset of) the given recording.
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
pub fn append(&mut self, db: &db::LockedDatabase, row: db::ListRecordingsRow,
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
rel_range_90k: Range<i32>) -> Result<(), Error> {
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
if let Some(prev) = self.segments.last() {
if prev.s.have_trailing_zero {
return Err(Error::new(format!(
"unable to append recording {}/{} after recording {}/{} with trailing zero",
row.camera_id, row.id, prev.s.camera_id, prev.s.recording_id)));
}
}
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
self.segments.push(Segment{
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
s: recording::Segment::new(db, &row, rel_range_90k)?,
shrink mp4::Segment 128 -> 112 bytes (on 64-bit) * don't store sizes of mp4-format sample indexes; recalculate them. * keep SampleIndexIterator position as a u32 rather than a usize. This is 960 bytes for a 60-minute mp4; another small cache usage improvement.
2017-02-26 03:02:49 -05:00
index: LazyCell::new(),
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
first_frame_num: self.next_frame_num,
num_subtitle_samples: 0,
});
self.next_frame_num += row.video_samples as u32;
if !self.video_sample_entries.iter().any(|e| e.id == row.video_sample_entry.id) {
self.video_sample_entries.push(row.video_sample_entry);
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
Ok(())
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
/// Builds the `File`, consuming the builder.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
pub fn build(mut self, db: Arc<db::Database>, dir: Arc<dir::SampleFileDir>)
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
-> Result<File, Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let mut max_end = None;
update several dependencies I left serde alone because uuid hasn't been updated for the new version.
2017-01-27 23:58:04 -05:00
let mut etag = hash::Hasher::new(hash::MessageDigest::sha1())?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
etag.update(&FORMAT_VERSION[..])?;
if self.include_timestamp_subtitle_track {
etag.update(b":ts:")?;
}
for s in &mut self.segments {
let d = &s.s.desired_range_90k;
self.duration_90k += (d.end - d.start) as u32;
let end = s.s.start + recording::Duration(d.end as i64);
max_end = match max_end {
None => Some(end),
Some(v) => Some(cmp::max(v, end)),
};
if self.include_timestamp_subtitle_track {
// Calculate the number of subtitle samples: starting to ending time (rounding up).
let start_sec = (s.s.start + recording::Duration(d.start as i64)).unix_seconds();
let end_sec = (s.s.start +
recording::Duration(d.end as i64 + TIME_UNITS_PER_SEC - 1))
.unix_seconds();
s.num_subtitle_samples = (end_sec - start_sec) as u32;
self.num_subtitle_samples += s.num_subtitle_samples;
}
// Update the etag to reflect this segment.
let mut data = [0_u8; 24];
let mut cursor = io::Cursor::new(&mut data[..]);
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
cursor.write_i32::<BigEndian>(s.s.camera_id)?;
cursor.write_i32::<BigEndian>(s.s.recording_id)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
cursor.write_i64::<BigEndian>(s.s.start.0)?;
cursor.write_i32::<BigEndian>(d.start)?;
cursor.write_i32::<BigEndian>(d.end)?;
etag.update(cursor.into_inner())?;
}
let max_end = match max_end {
None => return Err(Error::new("no segments!".to_owned())),
Some(v) => v,
};
let creation_ts = to_iso14496_timestamp(max_end);
let mut est_slices = 16 + self.video_sample_entries.len() + 4 * self.segments.len();
if self.include_timestamp_subtitle_track {
est_slices += 16 + self.segments.len();
}
self.body.slices.reserve(est_slices);
const EST_BUF_LEN: usize = 2048;
self.body.buf.reserve(EST_BUF_LEN);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.append_static(StaticBytestring::FtypBox)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
self.append_moov(creation_ts)?;
// Write the mdat header. Use the large format to support files over 2^32-1 bytes long.
// Write zeroes for the length as a placeholder; fill it in after it's known.
// It'd be nice to use the until-EOF form, but QuickTime Player doesn't support it.
self.body.buf.extend_from_slice(b"\x00\x00\x00\x01mdat\x00\x00\x00\x00\x00\x00\x00\x00");
let mdat_len_pos = self.body.buf.len() - 8;
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.flush_buf()?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let initial_sample_byte_pos = self.body.slices.len();
for (i, s) in self.segments.iter().enumerate() {
let r = s.s.sample_file_range();
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.append_slice(r.end - r.start, SliceType::VideoSampleData, i)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
if let Some(p) = self.subtitle_co64_pos {
BigEndian::write_u64(&mut self.body.buf[p .. p + 8], self.body.slices.len());
for (i, s) in self.segments.iter().enumerate() {
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.append_slice(
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
s.num_subtitle_samples as u64 *
(mem::size_of::<u16>() + SUBTITLE_LENGTH) as u64,
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
SliceType::SubtitleSampleData, i)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
}
// Fill in the length left as a placeholder above. Note the 16 here is the length
// of the mdat header.
BigEndian::write_u64(&mut self.body.buf[mdat_len_pos .. mdat_len_pos + 8],
16 + self.body.slices.len() - initial_sample_byte_pos);
if est_slices < self.body.slices.num() {
warn!("Estimated {} slices; actually were {} slices", est_slices,
self.body.slices.num());
} else {
debug!("Estimated {} slices; actually were {} slices", est_slices,
self.body.slices.num());
}
if EST_BUF_LEN < self.body.buf.len() {
warn!("Estimated {} buf bytes; actually were {}", EST_BUF_LEN, self.body.buf.len());
} else {
debug!("Estimated {} buf bytes; actually were {}", EST_BUF_LEN, self.body.buf.len());
}
debug!("slices: {:?}", self.body.slices);
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
Ok(File{
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
db: db,
dir: dir,
segments: self.segments,
slices: self.body.slices,
buf: self.body.buf,
video_sample_entries: self.video_sample_entries,
initial_sample_byte_pos: initial_sample_byte_pos,
last_modified: header::HttpDate(time::at(Timespec::new(max_end.unix_seconds(), 0))),
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-09 00:28:50 -05:00
etag: header::EntityTag::strong(strutil::hex(&etag.finish()?)),
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
})
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `MovieBox` (ISO/IEC 14496-12 section 8.2.1).
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_moov(&mut self, creation_ts: u32) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"moov");
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.append_mvhd(creation_ts)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
self.append_video_trak(creation_ts)?;
if self.include_timestamp_subtitle_track {
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.append_subtitle_trak(creation_ts)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `MovieHeaderBox` version 0 (ISO/IEC 14496-12 section 8.2.2).
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_mvhd(&mut self, creation_ts: u32) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"mvhd\x00\x00\x00\x00");
self.body.append_u32(creation_ts);
self.body.append_u32(creation_ts);
self.body.append_u32(TIME_UNITS_PER_SEC as u32);
let d = self.duration_90k;
self.body.append_u32(d);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.append_static(StaticBytestring::MvhdJunk)?;
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
let next_track_id = if self.include_timestamp_subtitle_track { 3 } else { 2 };
self.body.append_u32(next_track_id);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `TrackBox` (ISO/IEC 14496-12 section 8.3.1) suitable for video.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_video_trak(&mut self, creation_ts: u32) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"trak");
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.append_video_tkhd(creation_ts)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
self.maybe_append_video_edts()?;
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.append_video_mdia(creation_ts)?;
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `TrackBox` (ISO/IEC 14496-12 section 8.3.1) suitable for subtitles.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_subtitle_trak(&mut self, creation_ts: u32) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"trak");
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.append_subtitle_tkhd(creation_ts)?;
self.append_subtitle_mdia(creation_ts)?;
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `TrackHeaderBox` (ISO/IEC 14496-12 section 8.3.2) suitable for video.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_video_tkhd(&mut self, creation_ts: u32) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
// flags 7: track_enabled | track_in_movie | track_in_preview
self.body.buf.extend_from_slice(b"tkhd\x00\x00\x00\x07");
self.body.append_u32(creation_ts);
self.body.append_u32(creation_ts);
self.body.append_u32(1); // track_id
self.body.append_u32(0); // reserved
self.body.append_u32(self.duration_90k);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.append_static(StaticBytestring::TkhdJunk)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let width = self.video_sample_entries.iter().map(|e| e.width).max().unwrap();
let height = self.video_sample_entries.iter().map(|e| e.height).max().unwrap();
self.body.append_u32((width as u32) << 16);
self.body.append_u32((height as u32) << 16);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `TrackHeaderBox` (ISO/IEC 14496-12 section 8.3.2) suitable for subtitles.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_subtitle_tkhd(&mut self, creation_ts: u32) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
// flags 7: track_enabled | track_in_movie | track_in_preview
self.body.buf.extend_from_slice(b"tkhd\x00\x00\x00\x07");
self.body.append_u32(creation_ts);
self.body.append_u32(creation_ts);
self.body.append_u32(2); // track_id
self.body.append_u32(0); // reserved
self.body.append_u32(self.duration_90k);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.append_static(StaticBytestring::TkhdJunk)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
self.body.append_u32(0); // width, unused.
self.body.append_u32(0); // height, unused.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends an `EditBox` (ISO/IEC 14496-12 section 8.6.5) suitable for video, if necessary.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn maybe_append_video_edts(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
#[derive(Debug, Default)]
struct Entry {
segment_duration: u64,
media_time: u64,
};
let mut flushed: Vec<Entry> = Vec::new();
let mut unflushed: Entry = Default::default();
let mut cur_media_time: u64 = 0;
for s in &self.segments {
// The actual range may start before the desired range because it can only start on a
// key frame. This relationship should hold true:
// actual start <= desired start < desired end
let actual = s.s.actual_time_90k();
let skip = s.s.desired_range_90k.start - actual.start;
let keep = s.s.desired_range_90k.end - s.s.desired_range_90k.start;
assert!(skip >= 0 && keep > 0, "desired={}..{} actual={}..{}",
s.s.desired_range_90k.start, s.s.desired_range_90k.end,
actual.start, actual.end);
cur_media_time += skip as u64;
if unflushed.segment_duration + unflushed.media_time == cur_media_time {
unflushed.segment_duration += keep as u64;
} else {
if unflushed.segment_duration > 0 {
flushed.push(unflushed);
}
unflushed = Entry{
segment_duration: keep as u64,
media_time: cur_media_time,
};
}
cur_media_time += keep as u64;
}
if flushed.is_empty() && unflushed.media_time == 0 {
return Ok(()); // use implicit one-to-one mapping.
}
flushed.push(unflushed);
debug!("Using edit list: {:?}", flushed);
write_length!(self, {
self.body.buf.extend_from_slice(b"edts");
write_length!(self, {
// Use version 1 for 64-bit times.
self.body.buf.extend_from_slice(b"elst\x01\x00\x00\x00");
self.body.append_u32(flushed.len() as u32);
for e in &flushed {
self.body.append_u64(e.segment_duration);
self.body.append_u64(e.media_time);
// media_rate_integer + media_rate_fraction: both fixed at 1
self.body.buf.extend_from_slice(b"\x00\x01\x00\x01");
}
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})?;
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `MediaBox` (ISO/IEC 14496-12 section 8.4.1) suitable for video.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_video_mdia(&mut self, creation_ts: u32) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"mdia");
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.append_mdhd(creation_ts)?;
self.body.append_static(StaticBytestring::VideoHdlrBox)?;
self.append_video_minf()?;
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `MediaBox` (ISO/IEC 14496-12 section 8.4.1) suitable for subtitles.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_subtitle_mdia(&mut self, creation_ts: u32) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"mdia");
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.append_mdhd(creation_ts)?;
self.body.append_static(StaticBytestring::SubtitleHdlrBox)?;
self.append_subtitle_minf()?;
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `MediaHeaderBox` (ISO/IEC 14496-12 section 8.4.2.) suitable for either the video
/// or subtitle track.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_mdhd(&mut self, creation_ts: u32) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"mdhd\x00\x00\x00\x00");
self.body.append_u32(creation_ts);
self.body.append_u32(creation_ts);
self.body.append_u32(TIME_UNITS_PER_SEC as u32);
self.body.append_u32(self.duration_90k);
self.body.append_u32(0x55c40000); // language=und + pre_defined
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `MediaInformationBox` (ISO/IEC 14496-12 section 8.4.4) suitable for video.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_video_minf(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.append_static(StaticBytestring::VideoMinfJunk)?;
self.append_video_stbl()?;
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `MediaInformationBox` (ISO/IEC 14496-12 section 8.4.4) suitable for subtitles.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_subtitle_minf(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.append_static(StaticBytestring::SubtitleMinfJunk)?;
self.append_subtitle_stbl()?;
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `SampleTableBox` (ISO/IEC 14496-12 section 8.5.1) suitable for video.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_video_stbl(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"stbl");
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.append_video_stsd()?;
self.append_video_stts()?;
self.append_video_stsc()?;
self.append_video_stsz()?;
self.append_video_co64()?;
self.append_video_stss()?;
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `SampleTableBox` (ISO/IEC 14496-12 section 8.5.1) suitable for subtitles.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_subtitle_stbl(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.append_static(StaticBytestring::SubtitleStblJunk)?;
self.append_subtitle_stts()?;
self.append_subtitle_stsc()?;
self.append_subtitle_stsz()?;
self.append_subtitle_co64()?;
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `SampleDescriptionBox` (ISO/IEC 14496-12 section 8.5.2) suitable for video.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_video_stsd(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"stsd\x00\x00\x00\x00");
let n_entries = self.video_sample_entries.len() as u32;
self.body.append_u32(n_entries);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.flush_buf()?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
for (i, e) in self.video_sample_entries.iter().enumerate() {
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.append_slice(e.data.len() as u64, SliceType::VideoSampleEntry, i)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `TimeToSampleBox` (ISO/IEC 14496-12 section 8.6.1) suitable for video.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_video_stts(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"stts\x00\x00\x00\x00");
let mut entry_count = 0;
for s in &self.segments {
entry_count += s.s.frames as u32;
}
self.body.append_u32(entry_count);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.flush_buf()?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
for (i, s) in self.segments.iter().enumerate() {
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.append_slice(
2 * (mem::size_of::<u32>() as u64) * (s.s.frames as u64), SliceType::Stts, i)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `TimeToSampleBox` (ISO/IEC 14496-12 section 8.6.1) suitable for subtitles.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_subtitle_stts(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"stts\x00\x00\x00\x00");
let entry_count_pos = self.body.buf.len();
self.body.append_u32(0); // placeholder for entry_count
let mut entry_count = 0;
for s in &self.segments {
let r = &s.s.desired_range_90k;
let start = s.s.start + recording::Duration(r.start as i64);
let end = s.s.start + recording::Duration(r.end as i64);
let start_next_sec = recording::Time(
start.0 + TIME_UNITS_PER_SEC - (start.0 % TIME_UNITS_PER_SEC));
if end <= start_next_sec {
// Segment doesn't last past the next second.
entry_count += 1;
self.body.append_u32(1); // count
self.body.append_u32((end - start).0 as u32); // duration
} else {
// The first subtitle just lasts until the next second.
entry_count += 1;
self.body.append_u32(1); // count
self.body.append_u32((start_next_sec - start).0 as u32); // duration
// Then there are zero or more "interior" subtitles, one second each.
let end_prev_sec = recording::Time(end.0 - (end.0 % TIME_UNITS_PER_SEC));
if start_next_sec < end_prev_sec {
entry_count += 1;
let interior = (end_prev_sec - start_next_sec).0 / TIME_UNITS_PER_SEC;
self.body.append_u32(interior as u32); // count
self.body.append_u32(TIME_UNITS_PER_SEC as u32); // duration
}
// Then there's a final subtitle for the remaining fraction of a second.
entry_count += 1;
self.body.append_u32(1); // count
self.body.append_u32((end - end_prev_sec).0 as u32); // duration
}
}
BigEndian::write_u32(&mut self.body.buf[entry_count_pos .. entry_count_pos + 4],
entry_count);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `SampleToChunkBox` (ISO/IEC 14496-12 section 8.7.4) suitable for video.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_video_stsc(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"stsc\x00\x00\x00\x00");
self.body.append_u32(self.segments.len() as u32);
for (i, s) in self.segments.iter().enumerate() {
self.body.append_u32((i + 1) as u32);
self.body.append_u32(s.s.frames as u32);
// Write sample_description_index.
let i = self.video_sample_entries.iter().position(
|e| e.id == s.s.video_sample_entry_id).unwrap();
self.body.append_u32((i + 1) as u32);
}
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `SampleToChunkBox` (ISO/IEC 14496-12 section 8.7.4) suitable for subtitles.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_subtitle_stsc(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(
b"stsc\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01");
self.body.append_u32(self.num_subtitle_samples);
self.body.append_u32(1);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `SampleSizeBox` (ISO/IEC 14496-12 section 8.7.3) suitable for video.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_video_stsz(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"stsz\x00\x00\x00\x00\x00\x00\x00\x00");
let mut entry_count = 0;
for s in &self.segments {
entry_count += s.s.frames as u32;
}
self.body.append_u32(entry_count);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.flush_buf()?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
for (i, s) in self.segments.iter().enumerate() {
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.append_slice(
(mem::size_of::<u32>()) as u64 * (s.s.frames as u64), SliceType::Stsz, i)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `SampleSizeBox` (ISO/IEC 14496-12 section 8.7.3) suitable for subtitles.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_subtitle_stsz(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"stsz\x00\x00\x00\x00");
self.body.append_u32((mem::size_of::<u16>() + SUBTITLE_LENGTH) as u32);
self.body.append_u32(self.num_subtitle_samples);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `ChunkLargeOffsetBox` (ISO/IEC 14496-12 section 8.7.5) suitable for video.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_video_co64(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"co64\x00\x00\x00\x00");
self.body.append_u32(self.segments.len() as u32);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.flush_buf()?;
self.body.append_slice(
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
(mem::size_of::<u64>()) as u64 * (self.segments.len() as u64),
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
SliceType::Co64, 0)?;
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `ChunkLargeOffsetBox` (ISO/IEC 14496-12 section 8.7.5) suitable for subtitles.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_subtitle_co64(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
// Write a placeholder; the actual value will be filled in later.
self.body.buf.extend_from_slice(
b"co64\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00");
self.subtitle_co64_pos = Some(self.body.buf.len() - 8);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a `SyncSampleBox` (ISO/IEC 14496-12 section 8.6.2) suitable for video.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_video_stss(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
write_length!(self, {
self.body.buf.extend_from_slice(b"stss\x00\x00\x00\x00");
let mut entry_count = 0;
for s in &self.segments {
entry_count += s.s.key_frames as u32;
}
self.body.append_u32(entry_count);
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.flush_buf()?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
for (i, s) in self.segments.iter().enumerate() {
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.body.append_slice(
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
(mem::size_of::<u32>() as u64) * (s.s.key_frames as u64),
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
SliceType::Stss, i)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
}
impl BodyState {
fn append_u32(&mut self, v: u32) {
self.buf.write_u32::<BigEndian>(v).expect("Vec write shouldn't fail");
}
fn append_u64(&mut self, v: u64) {
self.buf.write_u64::<BigEndian>(v).expect("Vec write shouldn't fail");
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Flushes the buffer: appends a slice for everything written into the buffer so far,
/// noting the position which has been flushed. Call this method prior to adding any non-buffer
/// slice.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn flush_buf(&mut self) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let len = self.buf.len();
if self.unflushed_buf_pos < len {
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
let p = self.unflushed_buf_pos;
self.append_slice((len - p) as u64, SliceType::Buf, p)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
self.unflushed_buf_pos = len;
}
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
Ok(())
}
fn append_slice(&mut self, len: u64, t: SliceType, p: usize) -> Result<(), Error> {
let l = self.slices.len();
self.slices.append(Slice::new(l + len, t, p)?);
Ok(())
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Appends a static bytestring, flushing the buffer if necessary.
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn append_static(&mut self, which: StaticBytestring) -> Result<(), Error> {
self.flush_buf()?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let s = STATIC_BYTESTRINGS[which as usize];
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
self.append_slice(s.len() as u64, SliceType::Static, which as usize)
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
}
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
pub struct File {
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
db: Arc<db::Database>,
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
dir: Arc<dir::SampleFileDir>,
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
segments: Vec<Segment>,
use associated types for Slice This is more readable; in particular, it avoids the need for awkward the PhantomData in the Slices struct.
2017-02-25 21:54:52 -05:00
slices: Slices<Slice>,
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
buf: Vec<u8>,
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
video_sample_entries: SmallVec<[Arc<db::VideoSampleEntry>; 1]>,
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
initial_sample_byte_pos: u64,
last_modified: header::HttpDate,
etag: header::EntityTag,
}
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
impl File {
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn write_co64(&self, r: Range<u64>, l: u64, out: &mut io::Write) -> Result<(), Error> {
slices::clip_to_range(r, l, out, |w| {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let mut pos = self.initial_sample_byte_pos;
for s in &self.segments {
w.write_u64::<BigEndian>(pos)?;
let r = s.s.sample_file_range();
pos += r.end - r.start;
}
Ok(())
})
}
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn write_video_sample_data(&self, i: usize, r: Range<u64>, out: &mut io::Write)
-> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let s = &self.segments[i];
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
let rec = self.db.lock().get_recording_playback(s.s.camera_id, s.s.recording_id)?;
let f = self.dir.open_sample_file(rec.sample_file_uuid)?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
mmapfile::MmapFileSlice::new(f, s.s.sample_file_range()).write_to(r, out)
}
fn write_subtitle_sample_data(&self, i: usize, r: Range<u64>, l: u64, out: &mut io::Write)
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
-> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let s = &self.segments[i];
let d = &s.s.desired_range_90k;
let start_sec = (s.s.start + recording::Duration(d.start as i64)).unix_seconds();
let end_sec = (s.s.start + recording::Duration(d.end as i64 + TIME_UNITS_PER_SEC - 1))
.unix_seconds();
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
slices::clip_to_range(r, l, out, |w| {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
for ts in start_sec .. end_sec {
w.write_u16::<BigEndian>(SUBTITLE_LENGTH as u16)?;
let tm = time::at(time::Timespec{sec: ts, nsec: 0});
use std::io::Write;
write!(w, "{}", tm.strftime(SUBTITLE_TEMPLATE)?)?;
}
Ok(())
})?;
Ok(())
}
}
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
impl http_entity::Entity<Error> for File {
update to latest http-entity
2017-01-28 23:10:21 -05:00
fn add_headers(&self, hdrs: &mut header::Headers) {
hdrs.set(header::ContentType("video/mp4".parse().unwrap()));
}
fn last_modified(&self) -> Option<header::HttpDate> { Some(self.last_modified) }
fn etag(&self) -> Option<header::EntityTag> { Some(self.etag.clone()) }
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
fn len(&self) -> u64 { self.slices.len() }
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
fn write_to(&self, range: Range<u64>, out: &mut io::Write) -> Result<(), Error> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
self.slices.write_to(self, range, out)
}
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Tests. There are two general strategies used to validate the resulting files:
///
/// * basic tests that ffmpeg can read the generated mp4s. This ensures compatibility with
/// popular software, though it's hard to test specifics. ffmpeg provides an abstraction layer
/// over the encapsulation format, so mp4-specific details are hard to see. Also, ffmpeg might
/// paper over errors in the mp4 or have its own bugs.
///
/// * tests using the `BoxCursor` type to inspect the generated mp4s more closely. These don't
/// detect misunderstandings of the specification or incompatibilities, but they can be used
/// to verify the output is byte-for-byte as expected.
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
#[cfg(test)]
mod tests {
compile with stable Rust The benchmarks now require "cargo bench --features=nightly". The extra #[cfg(nightly)] switches in the code needed for it are a bit annoying; I may move the benches to a separate directory to avoid this. But for now, this works.
2016-12-10 01:04:35 -05:00
use byteorder::{BigEndian, ByteOrder};
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
use db;
use dir;
replace resource.rs with new http-entity crate This crate is a slightly-more-polished and MIT-licensed version of resource.rs. So far it has one advantage: running the tests doesn't require RUST_TEST_THREADS=1.
2016-12-20 21:29:45 -05:00
use error::Error;
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
use ffmpeg;
compile with stable Rust The benchmarks now require "cargo bench --features=nightly". The extra #[cfg(nightly)] switches in the code needed for it are a bit annoying; I may move the benches to a separate directory to avoid this. But for now, this works.
2016-12-10 01:04:35 -05:00
use hyper::header;
update several dependencies I left serde alone because uuid hasn't been updated for the new version.
2017-01-27 23:58:04 -05:00
use openssl::hash;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
use recording::{self, TIME_UNITS_PER_SEC};
replace resource.rs with new http-entity crate This crate is a slightly-more-polished and MIT-licensed version of resource.rs. So far it has one advantage: running the tests doesn't require RUST_TEST_THREADS=1.
2016-12-20 21:29:45 -05:00
use http_entity::{self, Entity};
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
use std::fs;
use std::io;
compile with stable Rust The benchmarks now require "cargo bench --features=nightly". The extra #[cfg(nightly)] switches in the code needed for it are a bit annoying; I may move the benches to a separate directory to avoid this. But for now, this works.
2016-12-10 01:04:35 -05:00
use std::ops::Range;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
use std::path::Path;
use std::sync::Arc;
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
use std::str;
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-09 00:28:50 -05:00
use strutil;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
use super::*;
add a basic test of Streamer, fix it This test is copied from the C++ implementation. It ensures the timestamps are calculated accurately from the pts rather than using ffmpeg's estimated duration. The Rust implementation was doing the easy-but-inaccurate thing, so fix that to make the test pass. Additionally, I did this with a code structure that should ensure the Rust code never drops a Writer without indicating to the syncer that its uuid is abandoned. Such a bug essentially leaks the partially-written file, although a restart would cause it to be properly unlinked and marked as such. There are no tests (yet) that exercise this scenario, though.
2016-12-06 21:41:44 -05:00
use stream::{self, Opener, Stream};
fix up the benchmarks These are currently the only thing which require a nightly Rust. I haven't run them since adding the feature gates. The feature gates were slightly broken, and the actual benchmarks had bitrotted a bit. Fix these things. Also put them into a separate submodule from the regular tests, so that not as many feature gates (#[cfg(feature="nightly")]) are required.
2017-01-08 17:22:35 -05:00
use testutil::{self, TestDb, TEST_CAMERA_ID};
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// A wrapper around openssl's SHA-1 hashing that implements the `Write` trait.
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
struct Sha1(hash::Hasher);
impl Sha1 {
update several dependencies I left serde alone because uuid hasn't been updated for the new version.
2017-01-27 23:58:04 -05:00
fn new() -> Sha1 { Sha1(hash::Hasher::new(hash::MessageDigest::sha1()).unwrap()) }
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
fn finish(mut self) -> Vec<u8> { self.0.finish().unwrap() }
}
impl io::Write for Sha1 {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
self.0.update(buf).unwrap();
Ok(buf.len())
}
fn flush(&mut self) -> io::Result<()> { Ok(()) }
}
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
/// Returns the SHA-1 digest of the given `Entity`.
fn digest(e: &http_entity::Entity<Error>) -> Vec<u8> {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let mut sha1 = Sha1::new();
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
e.write_to(0 .. e.len(), &mut sha1).unwrap();
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
sha1.finish()
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// Information used within `BoxCursor` to describe a box on the stack.
#[derive(Clone)]
struct Mp4Box {
interior: Range<u64>,
boxtype: [u8; 4],
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
/// A cursor over the boxes in a `.mp4` file. Supports moving forward and up/down the box
/// stack, not backward. Panics on error.
#[derive(Clone)]
struct BoxCursor<'a> {
replace resource.rs with new http-entity crate This crate is a slightly-more-polished and MIT-licensed version of resource.rs. So far it has one advantage: running the tests doesn't require RUST_TEST_THREADS=1.
2016-12-20 21:29:45 -05:00
mp4: &'a http_entity::Entity<Error>,
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
stack: Vec<Mp4Box>,
}
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
impl<'a> BoxCursor<'a> {
replace resource.rs with new http-entity crate This crate is a slightly-more-polished and MIT-licensed version of resource.rs. So far it has one advantage: running the tests doesn't require RUST_TEST_THREADS=1.
2016-12-20 21:29:45 -05:00
pub fn new(mp4: &'a http_entity::Entity<Error>) -> BoxCursor<'a> {
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
BoxCursor{
mp4: mp4,
stack: Vec::new(),
}
}
/// Pushes the box at the given position onto the stack (returning true), or returns
/// false if pos == max.
fn internal_push(&mut self, pos: u64, max: u64) -> bool {
if pos == max { return false; }
let mut hdr = [0u8; 16];
self.mp4.write_to(pos .. pos+8, &mut &mut hdr[..]).unwrap();
let (len, hdr_len, boxtype_slice) = match BigEndian::read_u32(&hdr[..4]) {
0 => (self.mp4.len() - pos, 8, &hdr[4..8]),
1 => {
self.mp4.write_to(pos+8 .. pos+12, &mut &mut hdr[..]).unwrap();
(BigEndian::read_u64(&hdr[4..12]), 16, &hdr[12..])
},
l => (l as u64, 8, &hdr[4..8]),
};
let mut boxtype = [0u8; 4];
assert!(pos + (hdr_len as u64) <= max);
assert!(pos + len <= max);
boxtype[..].copy_from_slice(boxtype_slice);
self.stack.push(Mp4Box{
interior: pos + hdr_len as u64 .. pos + len,
boxtype: boxtype,
});
trace!("positioned at {}", self.path());
true
}
fn path(&self) -> String {
let mut s = String::with_capacity(5 * self.stack.len());
for b in &self.stack {
s.push('/');
s.push_str(str::from_utf8(&b.boxtype[..]).unwrap());
}
s
}
/// Gets the specified byte range within the current box, starting after the box type.
/// Must not be at EOF.
pub fn get(&self, r: Range<u64>, mut buf: &mut [u8]) {
let interior = &self.stack.last().expect("at root").interior;
assert!(r.end < interior.end - interior.start);
self.mp4.write_to(r.start+interior.start .. r.end+interior.start, &mut buf).unwrap();
}
pub fn get_all(&self) -> Vec<u8> {
let interior = self.stack.last().expect("at root").interior.clone();
let len = (interior.end - interior.start) as usize;
let mut out = Vec::with_capacity(len);
self.mp4.write_to(interior, &mut out).unwrap();
out
}
/// Gets the specified u32 within the current box, starting after the box type.
/// Must not be at EOF.
pub fn get_u32(&self, p: u64) -> u32 {
let mut buf = [0u8; 4];
self.get(p .. p+4, &mut buf);
BigEndian::read_u32(&buf[..])
}
/// Navigates to the next box after the current one, or up if the current one is last.
pub fn next(&mut self) -> bool {
let old = self.stack.pop().expect("positioned at root; there is no next");
let max = self.stack.last().map(|b| b.interior.end).unwrap_or_else(|| self.mp4.len());
self.internal_push(old.interior.end, max)
}
/// Finds the next box of the given type after the current one, or navigates up if absent.
pub fn find(&mut self, boxtype: &[u8]) -> bool {
trace!("looking for {}", str::from_utf8(boxtype).unwrap());
loop {
if &self.stack.last().unwrap().boxtype[..] == boxtype {
return true;
}
if !self.next() {
return false;
}
}
}
/// Moves up the stack. Must not be at root.
pub fn up(&mut self) { self.stack.pop(); }
/// Moves down the stack. Must be positioned on a box with children.
pub fn down(&mut self) {
let range = self.stack.last().map(|b| b.interior.clone())
.unwrap_or_else(|| 0 .. self.mp4.len());
assert!(self.internal_push(range.start, range.end), "no children in {}", self.path());
}
}
/// Information returned by `find_track`.
struct Track<'a> {
edts_cursor: Option<BoxCursor<'a>>,
stbl_cursor: BoxCursor<'a>,
}
/// Finds the `moov/trak` that has a `tkhd` associated with the given `track_id`, which must
/// exist.
replace resource.rs with new http-entity crate This crate is a slightly-more-polished and MIT-licensed version of resource.rs. So far it has one advantage: running the tests doesn't require RUST_TEST_THREADS=1.
2016-12-20 21:29:45 -05:00
fn find_track(mp4: &http_entity::Entity<Error>, track_id: u32) -> Track {
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
let mut cursor = BoxCursor::new(mp4);
cursor.down();
assert!(cursor.find(b"moov"));
cursor.down();
loop {
assert!(cursor.find(b"trak"));
cursor.down();
assert!(cursor.find(b"tkhd"));
let mut version = [0u8; 1];
cursor.get(0 .. 1, &mut version);
// Let id_pos be the offset after the FullBox section of the track_id.
let id_pos = match version[0] {
0 => 8, // track_id follows 32-bit creation_time and modification_time
1 => 16, // ...64-bit times...
v => panic!("unexpected tkhd version {}", v),
};
let cur_track_id = cursor.get_u32(4 + id_pos);
trace!("found moov/trak/tkhd with id {}; want {}", cur_track_id, track_id);
if cur_track_id == track_id {
break;
}
cursor.up();
assert!(cursor.next());
}
let edts_cursor;
if cursor.find(b"edts") {
edts_cursor = Some(cursor.clone());
cursor.up();
} else {
edts_cursor = None;
};
cursor.down();
assert!(cursor.find(b"mdia"));
cursor.down();
assert!(cursor.find(b"minf"));
cursor.down();
assert!(cursor.find(b"stbl"));
Track{
edts_cursor: edts_cursor,
stbl_cursor: cursor,
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
}
fn copy_mp4_to_db(db: &TestDb) {
add a basic test of Streamer, fix it This test is copied from the C++ implementation. It ensures the timestamps are calculated accurately from the pts rather than using ffmpeg's estimated duration. The Rust implementation was doing the easy-but-inaccurate thing, so fix that to make the test pass. Additionally, I did this with a code structure that should ensure the Rust code never drops a Writer without indicating to the syncer that its uuid is abandoned. Such a bug essentially leaks the partially-written file, although a restart would cause it to be properly unlinked and marked as such. There are no tests (yet) that exercise this scenario, though.
2016-12-06 21:41:44 -05:00
let mut input =
stream::FFMPEG.open(stream::Source::File("src/testdata/clip.mp4")).unwrap();
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
// 2015-04-26 00:00:00 UTC.
const START_TIME: recording::Time = recording::Time(1430006400i64 * TIME_UNITS_PER_SEC);
let extra_data = input.get_extra_data().unwrap();
let video_sample_entry_id = db.db.lock().insert_video_sample_entry(
extra_data.width, extra_data.height, &extra_data.sample_entry).unwrap();
skip the first rotation This is as described in design/time.md.
2016-12-29 16:07:25 -05:00
let mut output = db.dir.create_writer(&db.syncer_channel, None,
add a basic test of Streamer, fix it This test is copied from the C++ implementation. It ensures the timestamps are calculated accurately from the pts rather than using ffmpeg's estimated duration. The Rust implementation was doing the easy-but-inaccurate thing, so fix that to make the test pass. Additionally, I did this with a code structure that should ensure the Rust code never drops a Writer without indicating to the syncer that its uuid is abandoned. Such a bug essentially leaks the partially-written file, although a restart would cause it to be properly unlinked and marked as such. There are no tests (yet) that exercise this scenario, though.
2016-12-06 21:41:44 -05:00
TEST_CAMERA_ID, video_sample_entry_id).unwrap();
// end_pts is the pts of the end of the most recent frame (start + duration).
// It's needed because dir::Writer calculates a packet's duration from its pts and the
// next packet's pts. That's more accurate for RTSP than ffmpeg's estimate of duration.
// To write the final packet of this sample .mp4 with a full duration, we need to fake a
// next packet's pts from the ffmpeg-supplied duration.
let mut end_pts = None;
new logic for calculating a recording's start time This is as described in design/time.md. Other aspects of that design (including using the monotonic clock and adjusting the durations to compensate for camera clock frequency error) are not implemented yet. No new tests yet. Just trying to get some flight miles on these ideas as soon as I can.
2016-12-28 23:56:08 -05:00
let mut frame_time = START_TIME;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
loop {
let pkt = match input.get_next() {
Ok(p) => p,
Err(ffmpeg::Error::Eof) => { break; },
Err(e) => { panic!("unexpected input error: {}", e); },
};
new logic for calculating a recording's start time This is as described in design/time.md. Other aspects of that design (including using the monotonic clock and adjusting the durations to compensate for camera clock frequency error) are not implemented yet. No new tests yet. Just trying to get some flight miles on these ideas as soon as I can.
2016-12-28 23:56:08 -05:00
let pts = pkt.pts().unwrap();
frame_time += recording::Duration(pkt.duration());
output.write(pkt.data().expect("packet without data"), frame_time, pts,
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
pkt.is_key()).unwrap();
new logic for calculating a recording's start time This is as described in design/time.md. Other aspects of that design (including using the monotonic clock and adjusting the durations to compensate for camera clock frequency error) are not implemented yet. No new tests yet. Just trying to get some flight miles on these ideas as soon as I can.
2016-12-28 23:56:08 -05:00
end_pts = Some(pts + pkt.duration());
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
add a basic test of Streamer, fix it This test is copied from the C++ implementation. It ensures the timestamps are calculated accurately from the pts rather than using ffmpeg's estimated duration. The Rust implementation was doing the easy-but-inaccurate thing, so fix that to make the test pass. Additionally, I did this with a code structure that should ensure the Rust code never drops a Writer without indicating to the syncer that its uuid is abandoned. Such a bug essentially leaks the partially-written file, although a restart would cause it to be properly unlinked and marked as such. There are no tests (yet) that exercise this scenario, though.
2016-12-06 21:41:44 -05:00
output.close(end_pts).unwrap();
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
db.syncer_channel.flush();
}
fix up the benchmarks These are currently the only thing which require a nightly Rust. I haven't run them since adding the feature gates. The feature gates were slightly broken, and the actual benchmarks had bitrotted a bit. Fix these things. Also put them into a separate submodule from the regular tests, so that not as many feature gates (#[cfg(feature="nightly")]) are required.
2017-01-08 17:22:35 -05:00
pub fn create_mp4_from_db(db: Arc<db::Database>, dir: Arc<dir::SampleFileDir>, skip_90k: i32,
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
shorten_90k: i32, include_subtitles: bool) -> File {
let mut builder = FileBuilder::new();
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
builder.include_timestamp_subtitle_track(include_subtitles);
let all_time = recording::Time(i64::min_value()) .. recording::Time(i64::max_value());
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
{
let db = db.lock();
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
db.list_recordings_by_time(TEST_CAMERA_ID, all_time, |r| {
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
let d = r.duration_90k;
assert!(skip_90k + shorten_90k < d);
builder.append(&db, r, skip_90k .. d - shorten_90k).unwrap();
Ok(())
}).unwrap();
}
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
builder.build(db, dir).unwrap()
}
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
fn write_mp4(mp4: &File, dir: &Path) -> String {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let mut filename = dir.to_path_buf();
filename.push("clip.new.mp4");
let mut out = fs::OpenOptions::new().write(true).create_new(true).open(&filename).unwrap();
mp4.write_to(0 .. mp4.len(), &mut out).unwrap();
filename.to_str().unwrap().to_string()
}
fn compare_mp4s(new_filename: &str, pts_offset: i64, shorten: i64) {
add a basic test of Streamer, fix it This test is copied from the C++ implementation. It ensures the timestamps are calculated accurately from the pts rather than using ffmpeg's estimated duration. The Rust implementation was doing the easy-but-inaccurate thing, so fix that to make the test pass. Additionally, I did this with a code structure that should ensure the Rust code never drops a Writer without indicating to the syncer that its uuid is abandoned. Such a bug essentially leaks the partially-written file, although a restart would cause it to be properly unlinked and marked as such. There are no tests (yet) that exercise this scenario, though.
2016-12-06 21:41:44 -05:00
let mut orig = stream::FFMPEG.open(stream::Source::File("src/testdata/clip.mp4")).unwrap();
let mut new = stream::FFMPEG.open(stream::Source::File(new_filename)).unwrap();
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
assert_eq!(orig.get_extra_data().unwrap(), new.get_extra_data().unwrap());
let mut final_durations = None;
loop {
let orig_pkt = match orig.get_next() {
Ok(p) => Some(p),
Err(ffmpeg::Error::Eof) => None,
Err(e) => { panic!("unexpected input error: {}", e); },
};
let new_pkt = match new.get_next() {
Ok(p) => Some(p),
Err(ffmpeg::Error::Eof) => { break; },
Err(e) => { panic!("unexpected input error: {}", e); },
};
let (orig_pkt, new_pkt) = match (orig_pkt, new_pkt) {
(Some(o), Some(n)) => (o, n),
(None, None) => break,
(o, n) => panic!("orig: {} new: {}", o.is_some(), n.is_some()),
};
assert_eq!(orig_pkt.pts().unwrap(), new_pkt.pts().unwrap() + pts_offset);
assert_eq!(orig_pkt.dts(), new_pkt.dts() + pts_offset);
assert_eq!(orig_pkt.data(), new_pkt.data());
assert_eq!(orig_pkt.is_key(), new_pkt.is_key());
final_durations = Some((orig_pkt.duration(), new_pkt.duration()));
}
if let Some((orig_dur, new_dur)) = final_durations {
// One would normally expect the duration to be exactly the same, but when using an
// edit list, ffmpeg appears to extend the last packet's duration by the amount skipped
// at the beginning. I think this is a bug on their side.
assert!(orig_dur - shorten + pts_offset == new_dur,
"orig_dur={} new_dur={} shorten={} pts_offset={}",
orig_dur, new_dur, shorten, pts_offset);
}
}
Shrink mp4 file slices from 16 to 8 bytes each For a one-hour recording, this is about 2 KiB, so a decent chunk of a Raspberry Pi 2's L1 cache. Generating the Slices and searching/scanning it should be a bit faster and pollute the cache less. This is a pretty small optimization now when transferring a decent chunk of the moov or mdat, but it's easy enough to do. It will be much more noticeable if I end up interleaving the captions between each key frame.
2017-02-21 22:37:36 -05:00
/// Makes a `.mp4` file which is only good for exercising the `Slice` logic for producing
/// sample tables that match the supplied encoder.
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
fn make_mp4_from_encoder(db: &TestDb, encoder: recording::SampleIndexEncoder,
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
desired_range_90k: Range<i32>) -> File {
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
let row = db.create_recording_from_encoder(encoder);
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-25 00:33:26 -05:00
let mut builder = FileBuilder::new();
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
builder.append(&db.db.lock(), row, desired_range_90k).unwrap();
builder.build(db.db.clone(), db.dir.clone()).unwrap()
}
/// Tests sample table for a simple video index of all sync frames.
#[test]
fn test_all_sync_frames() {
testutil::init();
let db = TestDb::new();
let mut encoder = recording::SampleIndexEncoder::new();
for i in 1..6 {
let duration_90k = 2 * i;
let bytes = 3 * i;
encoder.add_sample(duration_90k, bytes, true);
}
// Time range [2, 2+4+6+8) means the 2nd, 3rd, and 4th samples should be included.
let mp4 = make_mp4_from_encoder(&db, encoder, 2 .. 2+4+6+8);
let track = find_track(&mp4, 1);
assert!(track.edts_cursor.is_none());
let mut cursor = track.stbl_cursor;
cursor.down();
cursor.find(b"stts");
assert_eq!(cursor.get_all(), &[
0x00, 0x00, 0x00, 0x00, // version + flags
0x00, 0x00, 0x00, 0x03, // entry_count
// entries
0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x04, // run length / timestamps.
0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x06,
0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x08,
]);
cursor.find(b"stsz");
assert_eq!(cursor.get_all(), &[
0x00, 0x00, 0x00, 0x00, // version + flags
0x00, 0x00, 0x00, 0x00, // sample_size
0x00, 0x00, 0x00, 0x03, // sample_count
// entries
0x00, 0x00, 0x00, 0x06, // size
0x00, 0x00, 0x00, 0x09,
0x00, 0x00, 0x00, 0x0c,
]);
cursor.find(b"stss");
assert_eq!(cursor.get_all(), &[
0x00, 0x00, 0x00, 0x00, // version + flags
0x00, 0x00, 0x00, 0x03, // entry_count
// entries
0x00, 0x00, 0x00, 0x01, // sample_number
0x00, 0x00, 0x00, 0x02,
0x00, 0x00, 0x00, 0x03,
]);
}
/// Tests sample table and edit list for a video index with half sync frames.
#[test]
fn test_half_sync_frames() {
testutil::init();
let db = TestDb::new();
let mut encoder = recording::SampleIndexEncoder::new();
for i in 1..6 {
let duration_90k = 2 * i;
let bytes = 3 * i;
encoder.add_sample(duration_90k, bytes, (i % 2) == 1);
}
// Time range [2+4+6, 2+4+6+8) means the 4th sample should be included.
// The 3rd gets pulled in also because it's a sync frame and the 4th isn't.
let mp4 = make_mp4_from_encoder(&db, encoder, 2+4+6 .. 2+4+6+8);
let track = find_track(&mp4, 1);
// Examine edts. It should skip the 3rd frame.
let mut cursor = track.edts_cursor.unwrap();
cursor.down();
cursor.find(b"elst");
assert_eq!(cursor.get_all(), &[
0x01, 0x00, 0x00, 0x00, // version + flags
0x00, 0x00, 0x00, 0x01, // length
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, // segment_duration
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x06, // media_time
0x00, 0x01, 0x00, 0x01, // media_rate_{integer,fraction}
]);
// Examine stbl.
let mut cursor = track.stbl_cursor;
cursor.down();
cursor.find(b"stts");
assert_eq!(cursor.get_all(), &[
0x00, 0x00, 0x00, 0x00, // version + flags
0x00, 0x00, 0x00, 0x02, // entry_count
// entries
0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x06, // run length / timestamps.
0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x08,
]);
cursor.find(b"stsz");
assert_eq!(cursor.get_all(), &[
0x00, 0x00, 0x00, 0x00, // version + flags
0x00, 0x00, 0x00, 0x00, // sample_size
0x00, 0x00, 0x00, 0x02, // sample_count
// entries
0x00, 0x00, 0x00, 0x09, // size
0x00, 0x00, 0x00, 0x0c,
]);
cursor.find(b"stss");
assert_eq!(cursor.get_all(), &[
0x00, 0x00, 0x00, 0x00, // version + flags
0x00, 0x00, 0x00, 0x01, // entry_count
// entries
0x00, 0x00, 0x00, 0x01, // sample_number
]);
}
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
#[test]
fn test_round_trip() {
Make tests not care about the machine's timezone
2016-11-30 14:17:46 -05:00
testutil::init();
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
let db = TestDb::new();
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
copy_mp4_to_db(&db);
let mp4 = create_mp4_from_db(db.db.clone(), db.dir.clone(), 0, 0, false);
let new_filename = write_mp4(&mp4, db.tmpdir.path());
compare_mp4s(&new_filename, 0, 0);
// Test the metadata. This is brittle, which is the point. Any time the digest comparison
// here fails, it can be updated, but the etag must change as well! Otherwise clients may
// combine ranges from the new format with ranges from the old format.
let sha1 = digest(&mp4);
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-09 00:28:50 -05:00
assert_eq!("1e5331e8371bd97ac3158b3a86494abc87cdc70e", strutil::hex(&sha1[..]));
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
const EXPECTED_ETAG: &'static str = "908ae8ac303f66f2f4a1f8f52dba8f6ea9fdb442";
update to latest http-entity
2017-01-28 23:10:21 -05:00
assert_eq!(Some(header::EntityTag::strong(EXPECTED_ETAG.to_owned())), mp4.etag());
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
drop(db.syncer_channel);
db.syncer_join.join().unwrap();
}
#[test]
fn test_round_trip_with_subtitles() {
Make tests not care about the machine's timezone
2016-11-30 14:17:46 -05:00
testutil::init();
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
let db = TestDb::new();
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
copy_mp4_to_db(&db);
let mp4 = create_mp4_from_db(db.db.clone(), db.dir.clone(), 0, 0, true);
let new_filename = write_mp4(&mp4, db.tmpdir.path());
compare_mp4s(&new_filename, 0, 0);
// Test the metadata. This is brittle, which is the point. Any time the digest comparison
// here fails, it can be updated, but the etag must change as well! Otherwise clients may
// combine ranges from the new format with ranges from the old format.
let sha1 = digest(&mp4);
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-09 00:28:50 -05:00
assert_eq!("de382684a471f178e4e3a163762711b0653bfd83", strutil::hex(&sha1[..]));
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
const EXPECTED_ETAG: &'static str = "e21c6a6dfede1081db3701cc595ec267c43c2bff";
update to latest http-entity
2017-01-28 23:10:21 -05:00
assert_eq!(Some(header::EntityTag::strong(EXPECTED_ETAG.to_owned())), mp4.etag());
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
drop(db.syncer_channel);
db.syncer_join.join().unwrap();
}
#[test]
fn test_round_trip_with_edit_list() {
Make tests not care about the machine's timezone
2016-11-30 14:17:46 -05:00
testutil::init();
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
let db = TestDb::new();
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
copy_mp4_to_db(&db);
let mp4 = create_mp4_from_db(db.db.clone(), db.dir.clone(), 1, 0, false);
let new_filename = write_mp4(&mp4, db.tmpdir.path());
compare_mp4s(&new_filename, 1, 0);
// Test the metadata. This is brittle, which is the point. Any time the digest comparison
// here fails, it can be updated, but the etag must change as well! Otherwise clients may
// combine ranges from the new format with ranges from the old format.
let sha1 = digest(&mp4);
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-09 00:28:50 -05:00
assert_eq!("685e026af44204bc9cc52115c5e17058e9fb7c70", strutil::hex(&sha1[..]));
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
const EXPECTED_ETAG: &'static str = "1d5c5980f6ba08a4dd52dfd785667d42cdb16992";
update to latest http-entity
2017-01-28 23:10:21 -05:00
assert_eq!(Some(header::EntityTag::strong(EXPECTED_ETAG.to_owned())), mp4.etag());
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
drop(db.syncer_channel);
db.syncer_join.join().unwrap();
}
#[test]
fn test_round_trip_with_shorten() {
Make tests not care about the machine's timezone
2016-11-30 14:17:46 -05:00
testutil::init();
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
let db = TestDb::new();
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
copy_mp4_to_db(&db);
let mp4 = create_mp4_from_db(db.db.clone(), db.dir.clone(), 0, 1, false);
let new_filename = write_mp4(&mp4, db.tmpdir.path());
compare_mp4s(&new_filename, 0, 1);
// Test the metadata. This is brittle, which is the point. Any time the digest comparison
// here fails, it can be updated, but the etag must change as well! Otherwise clients may
// combine ranges from the new format with ranges from the old format.
let sha1 = digest(&mp4);
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-09 00:28:50 -05:00
assert_eq!("e0d28ddf08e24575a82657b1ce0b2da73f32fd88", strutil::hex(&sha1[..]));
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-21 01:08:18 -05:00
const EXPECTED_ETAG: &'static str = "555de64b39615e1a1cbe5bdd565ff197f5f126c5";
update to latest http-entity
2017-01-28 23:10:21 -05:00
assert_eq!(Some(header::EntityTag::strong(EXPECTED_ETAG.to_owned())), mp4.etag());
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
drop(db.syncer_channel);
db.syncer_join.join().unwrap();
}
fix up the benchmarks These are currently the only thing which require a nightly Rust. I haven't run them since adding the feature gates. The feature gates were slightly broken, and the actual benchmarks had bitrotted a bit. Fix these things. Also put them into a separate submodule from the regular tests, so that not as many feature gates (#[cfg(feature="nightly")]) are required.
2017-01-08 17:22:35 -05:00
}
#[cfg(all(test, feature="nightly"))]
mod bench {
extern crate test;
use hyper;
use hyper::header;
use http_entity;
use self::test::Bencher;
use std::str;
use super::tests::create_mp4_from_db;
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 23:37:03 -05:00
use testutil::{self, TestDb};
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
/// An HTTP server for benchmarking.
/// It's used as a singleton via `lazy_static!` for two reasons:
///
/// * to avoid running out of file descriptors. `#[bench]` functions apparently get called
/// many times as the number of iterations is tuned, and hyper servers
/// [can't be shut down](https://github.com/hyperium/hyper/issues/338), so
/// otherwise the default Ubuntu 16.04.1 ulimit of 1024 files is quickly exhausted.
/// * so that when getting a CPU profile of the benchmark, more of the profile focuses
/// on the HTTP serving rather than the setup.
///
/// Currently this only serves a single `.mp4` file but we could set up variations to benchmark
/// different scenarios: with/without subtitles and edit lists, different lengths, serving
/// different fractions of the file, etc.
struct BenchServer {
url: hyper::Url,
generated_len: u64,
}
impl BenchServer {
fn new() -> BenchServer {
let mut listener = hyper::net::HttpListener::new("127.0.0.1:0").unwrap();
use hyper::net::NetworkListener;
let addr = listener.local_addr().unwrap();
let server = hyper::Server::new(listener);
let url = hyper::Url::parse(
format!("http://{}:{}/", addr.ip(), addr.port()).as_str()).unwrap();
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
let db = TestDb::new();
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 23:37:03 -05:00
testutil::add_dummy_recordings_to_db(&db.db, 60);
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let mp4 = create_mp4_from_db(db.db.clone(), db.dir.clone(), 0, 0, false);
let p = mp4.initial_sample_byte_pos;
use std::thread::spawn;
spawn(move || {
use hyper::server::{Request, Response, Fresh};
let (db, dir) = (db.db.clone(), db.dir.clone());
let _ = server.handle(move |req: Request, res: Response<Fresh>| {
let mp4 = create_mp4_from_db(db.clone(), dir.clone(), 0, 0, false);
replace resource.rs with new http-entity crate This crate is a slightly-more-polished and MIT-licensed version of resource.rs. So far it has one advantage: running the tests doesn't require RUST_TEST_THREADS=1.
2016-12-20 21:29:45 -05:00
http_entity::serve(&mp4, &req, res).unwrap();
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
});
});
BenchServer{
url: url,
generated_len: p,
}
}
}
lazy_static! {
static ref SERVER: BenchServer = { BenchServer::new() };
}
/// Benchmarks serving the generated part of a `.mp4` file (up to the first byte from disk).
#[bench]
switch to a hyper vendor branch with Nagle fix There were Nagle's algorithm delays in both the "fresh_client" and "reuse_client" versions of the .mp4 serving benchmark. Now performance is much more consistent.
2017-02-26 22:05:05 -05:00
fn serve_generated_bytes(b: &mut Bencher) {
Make tests not care about the machine's timezone
2016-11-30 14:17:46 -05:00
testutil::init();
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let server = &*SERVER;
let p = server.generated_len;
let mut buf = Vec::with_capacity(p as usize);
b.bytes = p;
let client = hyper::Client::new();
switch to a hyper vendor branch with Nagle fix There were Nagle's algorithm delays in both the "fresh_client" and "reuse_client" versions of the .mp4 serving benchmark. Now performance is much more consistent.
2017-02-26 22:05:05 -05:00
let mut run = || {
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
let mut resp =
client.get(server.url.clone())
.header(header::Range::Bytes(vec![header::ByteRangeSpec::FromTo(0, p - 1)]))
.send()
.unwrap();
buf.clear();
use std::io::Read;
let size = resp.read_to_end(&mut buf).unwrap();
assert_eq!(p, size as u64);
switch to a hyper vendor branch with Nagle fix There were Nagle's algorithm delays in both the "fresh_client" and "reuse_client" versions of the .mp4 serving benchmark. Now performance is much more consistent.
2017-02-26 22:05:05 -05:00
};
run(); // warm.
b.iter(run);
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
}
#[bench]
fn mp4_construction(b: &mut Bencher) {
Make tests not care about the machine's timezone
2016-11-30 14:17:46 -05:00
testutil::init();
test and fix .mp4 generation code * new, more thorough tests based on a "BoxCursor" which navigates the resulting .mp4. This tests everything the C++ code was testing on Mp4SamplePieces. And it goes beyond: it tests the actual resulting .mp4 file, not some internal logic. * fix recording::Segment::foreach to properly handle a truncated ending. Before this was causing a panic. * get rid of the separate recording::Segment::init method. This was some of the first Rust I ever wrote, and I must have thought I couldn't loan it my locked database. I can, and that's more clean. Now Segments are never half-initialized. Less to test, less to go wrong. * fix recording::Segment::new to treat a trailing zero duration on a segment with a non-zero start in the same way as it does with a zero start. I'm still not sure what I'm doing makes sense, but at least it's not surprisingly inconsistent. * add separate, smaller tests of recording::Segment * address a couple TODOs in the .mp4 code and add missing comments * change a couple panics on database corruption into cleaner error returns * increment the etag version given the .mp4 output has changed
2016-12-02 23:40:55 -05:00
let db = TestDb::new();
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 23:37:03 -05:00
testutil::add_dummy_recordings_to_db(&db.db, 60);
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 17:34:00 -05:00
b.iter(|| {
create_mp4_from_db(db.db.clone(), db.dir.clone(), 0, 0, false);
});
}
}
Reference in New Issue Copy Permalink