MyFavMirrors/moonfire-nvr
1
0
Fork 0
mirror of https://github.com/scottlamb/moonfire-nvr.git synced 2025-01-15 08:45:01 -05:00
Code Issues Packages Projects Releases Wiki Activity
moonfire-nvr/src/web.rs

1445 lines
61 KiB
Rust
Raw Normal View History

Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
// This file is part of Moonfire NVR, a security camera digital video recorder.
// Copyright (C) 2016 Scott Lamb <slamb@slamb.org>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// In addition, as a special exception, the copyright holders give
// permission to link the code of portions of this program with the
// OpenSSL library under certain conditions as described in each
// individual source file, and distribute linked combinations including
// the two.
//
// You must obey the GNU General Public License in all respects for all
// of the code used other than OpenSSL. If you modify file(s) with this
// exception, you may extend this exception to your version of the
// file(s), but you are not obligated to do so. If you do not wish to do
// so, delete this exception statement from your version. If you delete
// this exception statement from all source files in the program, then
// also delete it here.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
use base::clock::Clocks;
switch to websocket for live stream (#59) The multipart stream / hanging GET approach worked in a prototype for a single stream, but Chrome has a per-host limit of six connections. If I try streaming all my cameras at once, I hit that limit. I can't open all the streams, much less additional connections to load init segments and such. Websockets apparently has a much higher limit of 256.
2020-02-28 21:41:31 -08:00
use base::{ErrorKind, bail_t, strutil};
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
use bytes::Bytes;
upgrade to 2018 Rust edition This is mostly just "cargo fix --edition" + Cargo.toml changes. There's one fix for upgrading to NLL in db/writer.rs: Writer::previously_opened wouldn't build with NLL because of a double-borrow the previous borrow checker somehow didn't catch. Restructure to avoid it. I'll put elective NLL changes in a following commit.
2018-12-28 12:21:49 -06:00
use crate::body::{Body, BoxedError};
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
use crate::json;
use crate::mp4;
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
use base64;
use bytes::{BufMut, BytesMut};
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
use core::borrow::Borrow;
use core::str::FromStr;
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
use db::{auth, recording};
use db::dir::SampleFileDir;
use failure::{Error, bail, format_err};
support multiple sample file directories This is still pretty basic support. There's no config UI support for renaming/moving the sample file directories after they are created, and no error checking that the files are still in the expected place. I can imagine sysadmins getting into trouble trying to change things. I hope to address at least some of that in a follow-up change to introduce a versioning/locking scheme that ensures databases and sample file dirs match in some way. A bonus change that kinda got pulled along for the ride: a dialog pops up in the config UI while a stream is being tested. The experience was pretty bad before; there was no indication the button worked at all until it was done, sometimes many seconds later.
2018-02-11 22:45:51 -08:00
use fnv::FnvHashMap;
switch to websocket for live stream (#59) The multipart stream / hanging GET approach worked in a prototype for a single stream, but Chrome has a per-host limit of six connections. If I try streaming all my cameras at once, I hit that limit. I can't open all the streams, much less additional connections to load init segments and such. Websockets apparently has a much higher limit of 256.
2020-02-28 21:41:31 -08:00
use futures::sink::SinkExt;
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
use futures::future::{self, Future, TryFutureExt};
switch to websocket for live stream (#59) The multipart stream / hanging GET approach worked in a prototype for a single stream, but Chrome has a per-host limit of six connections. If I try streaming all my cameras at once, I hit that limit. I can't open all the streams, much less additional connections to load init segments and such. Websockets apparently has a much higher limit of 256.
2020-02-28 21:41:31 -08:00
use futures::stream::StreamExt;
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
use http::{Request, Response, status::StatusCode};
crate rename: http-(entity|file) -> http-serve
2018-01-23 11:08:21 -08:00
use http_serve;
upgrade to hyper 0.12.x Just one (intentional) functional change---now the streamers start shutting down while the webserver shuts down gracefully.
2018-08-29 22:26:19 -07:00
use http::header::{self, HeaderValue};
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
use lazy_static::lazy_static;
use log::{debug, info, warn};
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
use regex::Regex;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
use serde_json;
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
use std::collections::HashMap;
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
use std::cmp;
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
use std::fs;
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
use std::net::IpAddr;
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-08 21:28:50 -08:00
use std::ops::Range;
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
use std::path::PathBuf;
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
use std::pin::Pin;
style improvements * remove stuttering: mp4::Mp4Foo -> mp4::Foo * stop using a &MutexGuard<Foo> where a &Foo will do
2017-02-24 21:33:26 -08:00
use std::sync::Arc;
switch to websocket for live stream (#59) The multipart stream / hanging GET approach worked in a prototype for a single stream, but Chrome has a per-host limit of six connections. If I try streaming all my cameras at once, I hit that limit. I can't open all the streams, much less additional connections to load init segments and such. Websockets apparently has a much higher limit of 256.
2020-02-28 21:41:31 -08:00
use tokio_tungstenite::tungstenite;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
use url::form_urlencoded;
use uuid::Uuid;
lazy_static! {
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
/// Regex used to parse the `s` query parameter to `view.mp4`.
/// As described in `design/api.md`, this is of the form
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
/// `START_ID[-END_ID][@OPEN_ID][.[REL_START_TIME]-[REL_END_TIME]]`.
static ref SEGMENTS_RE: Regex =
Regex::new(r"^(\d+)(-\d+)?(@\d+)?(?:\.(\d+)?-(\d+)?)?$").unwrap();
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
type BoxedFuture = Box<dyn Future<Output = Result<Response<Body>, BoxedError>> +
Sync + Send + 'static>;
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
#[derive(Debug, Eq, PartialEq)]
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
enum Path {
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
TopLevel, // "/api/"
Request, // "/api/request"
InitSegment([u8; 20], bool), // "/api/init/<sha1>.mp4{.txt}"
Camera(Uuid), // "/api/cameras/<uuid>/"
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
Signals, // "/api/signals"
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
StreamRecordings(Uuid, db::StreamType), // "/api/cameras/<uuid>/<type>/recordings"
StreamViewMp4(Uuid, db::StreamType, bool), // "/api/cameras/<uuid>/<type>/view.mp4{.txt}"
StreamViewMp4Segment(Uuid, db::StreamType, bool), // "/api/cameras/<uuid>/<type>/view.m4s{.txt}"
backend support for live stream (#59) This is so far completely untested, for use by a new UI prototype. It creates a new URL endpoint which sends one video/mp4 media segment per key frame, with the dependent frames included. This means there will be about one key frame interval of latency (typically about a second). This seems hard to avoid, as mentioned in issue #59.
2019-01-21 15:58:52 -08:00
StreamLiveMp4Segments(Uuid, db::StreamType), // "/api/cameras/<uuid>/<type>/live.m4s"
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
Login, // "/api/login"
Logout, // "/api/logout"
Static, // (anything that doesn't start with "/api/")
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
NotFound,
}
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
impl Path {
fn decode(path: &str) -> Self {
if !path.starts_with("/api/") {
return Path::Static;
}
let path = &path["/api".len()..];
if path == "/" {
return Path::TopLevel;
}
match path {
"/login" => return Path::Login,
"/logout" => return Path::Logout,
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
"/request" => return Path::Request,
"/signals" => return Path::Signals,
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
_ => {},
};
if path.starts_with("/init/") {
let (debug, path) = if path.ends_with(".txt") {
(true, &path[0 .. path.len() - 4])
} else {
(false, path)
};
if path.len() != 50 || !path.ends_with(".mp4") {
return Path::NotFound;
}
if let Ok(sha1) = strutil::dehex(&path.as_bytes()[6..46]) {
return Path::InitSegment(sha1, debug);
}
support segmented mp4s This is intended to support HTML5 Media Source Extensions, which I expect to be the most practical way to make a good web UI with a proper scrub bar and such. This feature has had very limited testing on Chrome and Firefox, and that was not entirely successful. More work is needed before it's usable, but this seems like a helpful progress checkpoint.
2017-10-01 15:29:22 -07:00
return Path::NotFound;
}
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
if !path.starts_with("/cameras/") {
return Path::NotFound;
support segmented mp4s This is intended to support HTML5 Media Source Extensions, which I expect to be the most practical way to make a good web UI with a proper scrub bar and such. This feature has had very limited testing on Chrome and Firefox, and that was not entirely successful. More work is needed before it's usable, but this seems like a helpful progress checkpoint.
2017-10-01 15:29:22 -07:00
}
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
let path = &path["/cameras/".len()..];
let slash = match path.find('/') {
None => { return Path::NotFound; },
Some(s) => s,
};
let uuid = &path[0 .. slash];
let path = &path[slash+1 .. ];
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
// TODO(slamb): require uuid to be in canonical format.
let uuid = match Uuid::parse_str(uuid) {
Ok(u) => u,
Err(_) => { return Path::NotFound },
};
schema version 2: support sub streams This allows each camera to have a main and a sub stream. Previously there was a field in the schema for the sub stream's url, but it didn't do anything. Now you can configure individual retention for main and sub streams. They show up grouped in the UI. No support for upgrading from schema version 1 yet.
2018-01-23 11:05:07 -08:00
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
if path.is_empty() {
return Path::Camera(uuid);
}
schema version 2: support sub streams This allows each camera to have a main and a sub stream. Previously there was a field in the schema for the sub stream's url, but it didn't do anything. Now you can configure individual retention for main and sub streams. They show up grouped in the UI. No support for upgrading from schema version 1 yet.
2018-01-23 11:05:07 -08:00
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
let slash = match path.find('/') {
None => { return Path::NotFound; },
Some(s) => s,
};
let (type_, path) = path.split_at(slash);
schema version 2: support sub streams This allows each camera to have a main and a sub stream. Previously there was a field in the schema for the sub stream's url, but it didn't do anything. Now you can configure individual retention for main and sub streams. They show up grouped in the UI. No support for upgrading from schema version 1 yet.
2018-01-23 11:05:07 -08:00
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
let type_ = match db::StreamType::parse(type_) {
None => { return Path::NotFound; },
Some(t) => t,
};
match path {
"/recordings" => Path::StreamRecordings(uuid, type_),
"/view.mp4" => Path::StreamViewMp4(uuid, type_, false),
"/view.mp4.txt" => Path::StreamViewMp4(uuid, type_, true),
"/view.m4s" => Path::StreamViewMp4Segment(uuid, type_, false),
"/view.m4s.txt" => Path::StreamViewMp4Segment(uuid, type_, true),
backend support for live stream (#59) This is so far completely untested, for use by a new UI prototype. It creates a new URL endpoint which sends one video/mp4 media segment per key frame, with the dependent frames included. This means there will be about one key frame interval of latency (typically about a second). This seems hard to avoid, as mentioned in issue #59.
2019-01-21 15:58:52 -08:00
"/live.m4s" => Path::StreamLiveMp4Segments(uuid, type_),
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
_ => Path::NotFound,
}
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
fn plain_response<B: Into<Body>>(status: http::StatusCode, body: B) -> Response<Body> {
Response::builder()
.status(status)
.header(header::CONTENT_TYPE, HeaderValue::from_static("text/plain"))
.body(body.into()).expect("hardcoded head should be valid")
}
fn not_found<B: Into<Body>>(body: B) -> Response<Body> {
plain_response(StatusCode::NOT_FOUND, body)
}
fn bad_req<B: Into<Body>>(body: B) -> Response<Body> {
plain_response(StatusCode::BAD_REQUEST, body)
}
fn internal_server_err<E: Into<Error>>(err: E) -> Response<Body> {
plain_response(StatusCode::INTERNAL_SERVER_ERROR, err.into().to_string())
}
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
fn from_base_error(err: base::Error) -> Response<Body> {
let status_code = match err.kind() {
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
ErrorKind::PermissionDenied | ErrorKind::Unauthenticated => StatusCode::UNAUTHORIZED,
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
ErrorKind::InvalidArgument => StatusCode::BAD_REQUEST,
ErrorKind::NotFound => StatusCode::NOT_FOUND,
_ => StatusCode::INTERNAL_SERVER_ERROR,
};
plain_response(status_code, err.to_string())
}
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
#[derive(Debug, Eq, PartialEq)]
struct Segments {
ids: Range<i32>,
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
open_id: Option<u32>,
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
start_time: i64,
end_time: Option<i64>,
}
impl Segments {
pub fn parse(input: &str) -> Result<Segments, ()> {
let caps = SEGMENTS_RE.captures(input).ok_or(())?;
add matching time parsing and formatting routines * add a --ts subcommand to convert between numeric and human-readable representations. This is handy when directly inspecting the SQLite database or API output. * also take the human-readable form in the web interface's camera view. * to reduce confusion, when using trim=true on the web interface's camera view, trim the displayed starting and ending times as well as the actual .mp4 file links.
2017-01-12 23:09:02 -08:00
let ids_start = i32::from_str(caps.get(1).unwrap().as_str()).map_err(|_| ())?;
let ids_end = match caps.get(2) {
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
Some(m) => i32::from_str(&m.as_str()[1..]).map_err(|_| ())?,
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
None => ids_start,
} + 1;
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
let open_id = match caps.get(3) {
Some(m) => Some(u32::from_str(&m.as_str()[1..]).map_err(|_| ())?),
None => None,
};
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
if ids_start < 0 || ids_end <= ids_start {
return Err(());
}
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
let start_time = caps.get(4).map_or(Ok(0), |m| i64::from_str(m.as_str())).map_err(|_| ())?;
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
if start_time < 0 {
return Err(());
}
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
let end_time = match caps.get(5) {
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
Some(v) => {
add matching time parsing and formatting routines * add a --ts subcommand to convert between numeric and human-readable representations. This is handy when directly inspecting the SQLite database or API output. * also take the human-readable form in the web interface's camera view. * to reduce confusion, when using trim=true on the web interface's camera view, trim the displayed starting and ending times as well as the actual .mp4 file links.
2017-01-12 23:09:02 -08:00
let e = i64::from_str(v.as_str()).map_err(|_| ())?;
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
if e <= start_time {
return Err(());
}
Some(e)
},
None => None
};
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
Ok(Segments {
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
ids: ids_start .. ids_end,
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
open_id,
start_time,
end_time,
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
})
}
}
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
/// A user interface file (.html, .js, etc).
/// The list of files is loaded into the server at startup; this makes path canonicalization easy.
/// The files themselves are opened on every request so they can be changed during development.
#[derive(Debug)]
struct UiFile {
upgrade to hyper 0.12.x Just one (intentional) functional change---now the streamers start shutting down while the webserver shuts down gracefully.
2018-08-29 22:26:19 -07:00
mime: HeaderValue,
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
path: PathBuf,
}
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
struct Caller {
permissions: db::Permissions,
session: Option<json::Session>,
}
impl Caller {
}
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
struct ServiceInner {
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
db: Arc<db::Database>,
support multiple sample file directories This is still pretty basic support. There's no config UI support for renaming/moving the sample file directories after they are created, and no error checking that the files are still in the expected place. I can imagine sysadmins getting into trouble trying to change things. I hope to address at least some of that in a follow-up change to introduce a versioning/locking scheme that ensures databases and sample file dirs match in some way. A bonus change that kinda got pulled along for the ride: a dialog pops up in the config UI while a stream is being tested. The experience was pretty bad before; there was no indication the button worked at all until it was done, sometimes many seconds later.
2018-02-11 22:45:51 -08:00
dirs_by_stream_id: Arc<FnvHashMap<i32, Arc<SampleFileDir>>>,
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
ui_files: HashMap<String, UiFile>,
populate timeZoneName as expected by UI This works by a nasty hack, but it seems to work well enough for now. Fingers crossed.
2017-10-21 23:57:13 -07:00
time_zone_name: String,
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
allow_unauthenticated_permissions: Option<db::Permissions>,
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
trust_forward_hdrs: bool,
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
type ResponseResult = Result<Response<Body>, Response<Body>>;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
fn serve_json<T: serde::ser::Serialize>(req: &Request<hyper::Body>, out: &T) -> ResponseResult {
let (mut resp, writer) = http_serve::streaming_body(&req).build();
resp.headers_mut().insert(header::CONTENT_TYPE,
HeaderValue::from_static("application/json"));
if let Some(mut w) = writer {
serde_json::to_writer(&mut w, out).map_err(internal_server_err)?;
}
Ok(resp)
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
impl ServiceInner {
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
fn top_level(&self, req: &Request<::hyper::Body>, caller: Caller) -> ResponseResult {
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
let mut days = false;
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
let mut camera_configs = false;
gzip json responses, handle HEAD properly
2018-01-23 11:22:23 -08:00
if let Some(q) = req.uri().query() {
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
for (key, value) in form_urlencoded::parse(q.as_bytes()) {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let (key, value): (_, &str) = (key.borrow(), value.borrow());
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
match key {
"days" => days = value == "true",
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
"cameraConfigs" => camera_configs = value == "true",
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
_ => {},
};
}
}
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
if camera_configs {
if !caller.permissions.read_camera_configs {
return Err(plain_response(StatusCode::UNAUTHORIZED,
"read_camera_configs required"));
}
}
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
let db = self.db.lock();
serve_json(req, &json::TopLevel {
time_zone_name: &self.time_zone_name,
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
cameras: (&db, days, camera_configs),
session: caller.session,
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
signals: (&db, days),
signal_types: &db,
})
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
fn camera(&self, req: &Request<::hyper::Body>, uuid: Uuid) -> ResponseResult {
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
let db = self.db.lock();
let camera = db.get_camera(uuid)
.ok_or_else(|| not_found(format!("no such camera {}", uuid)))?;
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
serve_json(req, &json::Camera::wrap(camera, &db, true, false).map_err(internal_server_err)?)
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
upgrade to hyper 0.12.x Just one (intentional) functional change---now the streamers start shutting down while the webserver shuts down gracefully.
2018-08-29 22:26:19 -07:00
fn stream_recordings(&self, req: &Request<::hyper::Body>, uuid: Uuid, type_: db::StreamType)
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
-> ResponseResult {
support run splitting in json api
2017-10-17 09:00:05 -07:00
let (r, split) = {
db crate support for updating signals (#28) This is a definite work in progress. In particular, * there's no src/web.rs support yet so it can't be used, * the code is surprisingly complex, and there's almost no tests so far. I want to at least get complete branch coverage. * I may still go back to time_sec rather than time_90k to save RAM and flash. I simplified the approach a bit from the earlier goal in design/api.md. In particular, there's no longer the separate concept of "observation" vs "prediction". Now the predictions are just observations that extend a bit beyond now. They may be flushed prematurely and I'll try living with that to avoid making things even more complex.
2019-06-13 21:55:15 -07:00
let mut time = recording::Time::min_value() .. recording::Time::max_value();
support run splitting in json api
2017-10-17 09:00:05 -07:00
let mut split = recording::Duration(i64::max_value());
gzip json responses, handle HEAD properly
2018-01-23 11:22:23 -08:00
if let Some(q) = req.uri().query() {
support run splitting in json api
2017-10-17 09:00:05 -07:00
for (key, value) in form_urlencoded::parse(q.as_bytes()) {
let (key, value) = (key.borrow(), value.borrow());
match key {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
"startTime90k" => {
time.start = recording::Time::parse(value)
.map_err(|_| bad_req("unparseable startTime90k"))?
},
"endTime90k" => {
time.end = recording::Time::parse(value)
.map_err(|_| bad_req("unparseable endTime90k"))?
},
"split90k" => {
split = recording::Duration(i64::from_str(value)
.map_err(|_| bad_req("unparseable split90k"))?)
},
support run splitting in json api
2017-10-17 09:00:05 -07:00
_ => {},
}
};
}
(time, split)
};
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-08 21:28:50 -08:00
let mut out = json::ListRecordings{recordings: Vec::new()};
{
let db = self.db.lock();
let camera = db.get_camera(uuid)
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.ok_or_else(|| plain_response(StatusCode::NOT_FOUND,
format!("no such camera {}", uuid)))?;
schema version 2: support sub streams This allows each camera to have a main and a sub stream. Previously there was a field in the schema for the sub stream's url, but it didn't do anything. Now you can configure individual retention for main and sub streams. They show up grouped in the UI. No support for upgrading from schema version 1 yet.
2018-01-23 11:05:07 -08:00
let stream_id = camera.streams[type_.index()]
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.ok_or_else(|| plain_response(StatusCode::NOT_FOUND,
format!("no such stream {}/{}", uuid, type_)))?;
take FnMut closures by reference I mistakenly thought these had to be monomorphized. (The FnOnce still does, until rust-lang/rfcs#1909 is implemented.) Turns out this way works fine. It should result in less compile time / code size, though I didn't check this.
2018-02-23 09:19:42 -08:00
db.list_aggregated_recordings(stream_id, r, split, &mut |row| {
bugfix: in /recordings, end_id should be inclusive
2017-10-04 06:36:30 -07:00
let end = row.ids.end - 1; // in api, ids are inclusive.
move list_recordings_by_* logic into raw.rs I want to start having the db.rs version augment this with the uncommitted recordings, and it's nice to have the separation of the raw db vs augmented versions. Also, this fits with the general theme of shrinking db.rs a bit. I had to put the raw video_sample_entry_id into the rows rather than the video_sample_entry Arc. In hindsight, this is better anyway: the common callers don't need to do the btree lookup and arc clone on every row. I think I'd originally done it that way only because I was quite new to rust and didn't understand that db could be used from within the row callback given that both borrows are immutable.
2018-03-01 20:59:05 -08:00
let vse = db.video_sample_entries_by_id().get(&row.video_sample_entry_id).unwrap();
add start_id and end_id to .../recordings json This was added to the API documentation in eee887b9 but never actually implemented then. It's necessary to actually fetch the .mp4 in question.
2017-10-04 00:00:56 -07:00
out.recordings.push(json::Recording {
start_id: row.ids.start,
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
end_id: if end == row.ids.start { None } else { Some(end) },
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
start_time_90k: row.time.start.0,
end_time_90k: row.time.end.0,
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-08 21:28:50 -08:00
sample_file_bytes: row.sample_file_bytes,
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
open_id: row.open_id,
first_uncommitted: row.first_uncommitted,
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-08 21:28:50 -08:00
video_samples: row.video_samples,
move list_recordings_by_* logic into raw.rs I want to start having the db.rs version augment this with the uncommitted recordings, and it's nice to have the separation of the raw db vs augmented versions. Also, this fits with the general theme of shrinking db.rs a bit. I had to put the raw video_sample_entry_id into the rows rather than the video_sample_entry Arc. In hindsight, this is better anyway: the common callers don't need to do the btree lookup and arc clone on every row. I think I'd originally done it that way only because I was quite new to rust and didn't understand that db could be used from within the row callback given that both borrows are immutable.
2018-03-01 20:59:05 -08:00
video_sample_entry_width: vse.width,
video_sample_entry_height: vse.height,
video_sample_entry_sha1: strutil::hex(&vse.sha1),
view in-progress recordings! The time from recorded to viewable was previously 60-120 sec for the first recording of a RTSP session, 0-60 sec otherwise. Now it's one frame.
2018-03-02 15:40:32 -08:00
growing: row.growing,
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-08 21:28:50 -08:00
});
Ok(())
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}).map_err(internal_server_err)?;
fully implement json handling as in spec This is a significant milestone; now the Rust branch matches the C++ branch's features. In the process, I switched from using serde_derive (which requires nightly Rust) to serde_codegen (which does not). It was easier than I thought it'd be. I'm getting close to no longer requiring nightly Rust.
2016-12-08 21:28:50 -08:00
}
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
serve_json(req, &out)
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
fn init_segment(&self, sha1: [u8; 20], debug: bool, req: &Request<::hyper::Body>)
-> ResponseResult {
support segmented mp4s This is intended to support HTML5 Media Source Extensions, which I expect to be the most practical way to make a good web UI with a proper scrub bar and such. This feature has had very limited testing on Chrome and Firefox, and that was not entirely successful. More work is needed before it's usable, but this seems like a helpful progress checkpoint.
2017-10-01 15:29:22 -07:00
let mut builder = mp4::FileBuilder::new(mp4::Type::InitSegment);
let db = self.db.lock();
move list_recordings_by_* logic into raw.rs I want to start having the db.rs version augment this with the uncommitted recordings, and it's nice to have the separation of the raw db vs augmented versions. Also, this fits with the general theme of shrinking db.rs a bit. I had to put the raw video_sample_entry_id into the rows rather than the video_sample_entry Arc. In hindsight, this is better anyway: the common callers don't need to do the btree lookup and arc clone on every row. I think I'd originally done it that way only because I was quite new to rust and didn't understand that db could be used from within the row callback given that both borrows are immutable.
2018-03-01 20:59:05 -08:00
for ent in db.video_sample_entries_by_id().values() {
support segmented mp4s This is intended to support HTML5 Media Source Extensions, which I expect to be the most practical way to make a good web UI with a proper scrub bar and such. This feature has had very limited testing on Chrome and Firefox, and that was not entirely successful. More work is needed before it's usable, but this seems like a helpful progress checkpoint.
2017-10-01 15:29:22 -07:00
if ent.sha1 == sha1 {
builder.append_video_sample_entry(ent.clone());
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mp4 = builder.build(self.db.clone(), self.dirs_by_stream_id.clone())
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
.map_err(from_base_error)?;
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
if debug {
return Ok(plain_response(StatusCode::OK, format!("{:#?}", mp4)));
} else {
return Ok(http_serve::serve(mp4, req));
}
support segmented mp4s This is intended to support HTML5 Media Source Extensions, which I expect to be the most practical way to make a good web UI with a proper scrub bar and such. This feature has had very limited testing on Chrome and Firefox, and that was not entirely successful. More work is needed before it's usable, but this seems like a helpful progress checkpoint.
2017-10-01 15:29:22 -07:00
}
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
Err(not_found("no such init segment"))
support segmented mp4s This is intended to support HTML5 Media Source Extensions, which I expect to be the most practical way to make a good web UI with a proper scrub bar and such. This feature has had very limited testing on Chrome and Firefox, and that was not entirely successful. More work is needed before it's usable, but this seems like a helpful progress checkpoint.
2017-10-01 15:29:22 -07:00
}
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
fn stream_view_mp4(&self, req: &Request<::hyper::Body>, caller: Caller, uuid: Uuid,
backend support for live stream (#59) This is so far completely untested, for use by a new UI prototype. It creates a new URL endpoint which sends one video/mp4 media segment per key frame, with the dependent frames included. This means there will be about one key frame interval of latency (typically about a second). This seems hard to avoid, as mentioned in issue #59.
2019-01-21 15:58:52 -08:00
stream_type: db::StreamType, mp4_type: mp4::Type, debug: bool)
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
-> ResponseResult {
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
if !caller.permissions.view_video {
return Err(plain_response(StatusCode::UNAUTHORIZED, "view_video required"));
}
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
let (stream_id, camera_name);
{
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
let db = self.db.lock();
style: convert try!(...) to ...? in web.rs
2016-12-02 21:46:31 -08:00
let camera = db.get_camera(uuid)
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.ok_or_else(|| plain_response(StatusCode::NOT_FOUND,
format!("no such camera {}", uuid)))?;
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
camera_name = camera.short_name.clone();
stream_id = camera.streams[stream_type.index()]
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.ok_or_else(|| plain_response(StatusCode::NOT_FOUND,
format!("no such stream {}/{}", uuid,
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
stream_type)))?;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
};
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
let mut start_time_for_filename = None;
backend support for live stream (#59) This is so far completely untested, for use by a new UI prototype. It creates a new URL endpoint which sends one video/mp4 media segment per key frame, with the dependent frames included. This means there will be about one key frame interval of latency (typically about a second). This seems hard to avoid, as mentioned in issue #59.
2019-01-21 15:58:52 -08:00
let mut builder = mp4::FileBuilder::new(mp4_type);
schema version 2: support sub streams This allows each camera to have a main and a sub stream. Previously there was a field in the schema for the sub stream's url, but it didn't do anything. Now you can configure individual retention for main and sub streams. They show up grouped in the UI. No support for upgrading from schema version 1 yet.
2018-01-23 11:05:07 -08:00
if let Some(q) = req.uri().query() {
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
for (key, value) in form_urlencoded::parse(q.as_bytes()) {
let (key, value) = (key.borrow(), value.borrow());
match key {
"s" => {
let s = Segments::parse(value).map_err(
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
|()| plain_response(StatusCode::BAD_REQUEST,
format!("invalid s parameter: {}", value)))?;
schema version 2: support sub streams This allows each camera to have a main and a sub stream. Previously there was a field in the schema for the sub stream's url, but it didn't do anything. Now you can configure individual retention for main and sub streams. They show up grouped in the UI. No support for upgrading from schema version 1 yet.
2018-01-23 11:05:07 -08:00
debug!("stream_view_mp4: appending s={:?}", s);
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
let mut est_segments = (s.ids.end - s.ids.start) as usize;
if let Some(end) = s.end_time {
// There should be roughly ceil((end - start) /
// desired_recording_duration) recordings in the desired timespan if
// there are no gaps or overlap, possibly another for misalignment of
// the requested timespan with the rotate offset and another because
// rotation only happens at key frames.
let ceil_durations = (end - s.start_time +
recording::DESIRED_RECORDING_DURATION - 1) /
recording::DESIRED_RECORDING_DURATION;
est_segments = cmp::min(est_segments, (ceil_durations + 2) as usize);
}
builder.reserve(est_segments);
let db = self.db.lock();
let mut prev = None;
let mut cur_off = 0;
take FnMut closures by reference I mistakenly thought these had to be monomorphized. (The FnOnce still does, until rust-lang/rfcs#1909 is implemented.) Turns out this way works fine. It should result in less compile time / code size, though I didn't check this.
2018-02-23 09:19:42 -08:00
db.list_recordings_by_id(stream_id, s.ids.clone(), &mut |r| {
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 10:11:10 -08:00
let recording_id = r.id.recording();
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
if let Some(o) = s.open_id {
if r.open_id != o {
bail!("recording {} has open id {}, requested {}",
r.id, r.open_id, o);
}
}
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
// Check for missing recordings.
match prev {
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 10:11:10 -08:00
None if recording_id == s.ids.start => {},
replace homegrown Error with failure crate This reduces boilerplate, making it a bit easier for me to split the db stuff out into its own crate.
2018-02-20 22:46:14 -08:00
None => bail!("no such recording {}/{}", stream_id, s.ids.start),
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 10:11:10 -08:00
Some(id) if r.id.recording() != id + 1 => {
replace homegrown Error with failure crate This reduces boilerplate, making it a bit easier for me to split the db stuff out into its own crate.
2018-02-20 22:46:14 -08:00
bail!("no such recording {}/{}", stream_id, id + 1);
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
},
_ => {},
};
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 10:11:10 -08:00
prev = Some(recording_id);
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
// Add a segment for the relevant part of the recording, if any.
let end_time = s.end_time.unwrap_or(i64::max_value());
let d = r.duration_90k as i64;
if s.start_time <= cur_off + d && cur_off < end_time {
let start = cmp::max(0, s.start_time - cur_off);
let end = cmp::min(d, end_time - cur_off);
let times = start as i32 .. end as i32;
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 10:11:10 -08:00
debug!("...appending recording {} with times {:?} \
(out of dur {})", r.id, times, d);
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
if start_time_for_filename.is_none() {
start_time_for_filename =
Some(r.start + recording::Duration(start));
}
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
builder.append(&db, r, start as i32 .. end as i32)?;
} else {
reorganize the sample file directory The filenames now represent composite ids (stream id + recording id) rather than a separate uuid system with its own reservation for a few benefits: * This provides more information when there are inconsistencies. * This avoids the need for managing the reservations during recording. I expect this to simplify delaying flushing of newly written sample files. Now the directory has to be scanned at startup for files that never got written to the database, but that's acceptably fast even with millions of files. * Less information to keep in memory and in the recording_playback table. I'd considered using one directory per stream, which might help if the filesystem has trouble coping with huge directories. But that would mean each dir has to be fsync()ed separately (more latency and/or more multithreading). So I'll stick with this until I see concrete evidence of a problem that would solve. Test coverage of the error conditions is poor. I plan to do some restructuring of the db/dir code, hopefully making steps toward testability along the way.
2018-02-20 10:11:10 -08:00
debug!("...skipping recording {} dur {}", r.id, d);
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
}
cur_off += d;
Ok(())
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}).map_err(internal_server_err)?;
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
// Check for missing recordings.
match prev {
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
Some(id) if s.ids.end != id + 1 => {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
return Err(not_found(format!("no such recording {}/{}",
stream_id, s.ids.end - 1)));
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
},
None => {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
return Err(not_found(format!("no such recording {}/{}",
stream_id, s.ids.start)));
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
},
_ => {},
};
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
if let Some(end) = s.end_time {
if end > cur_off {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
return Err(plain_response(
StatusCode::BAD_REQUEST,
format!("end time {} is beyond specified recordings",
end)));
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
}
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
}
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
},
"ts" => builder.include_timestamp_subtitle_track(value == "true"),
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
_ => return Err(bad_req(format!("parameter {} not understood", key))),
upgrade to async hyper serve_generated_bytes is >3X faster. One caveat is that the reactor thread may stall when reading from the memory-mapped slice. Moonfire NVR is basically a single-user program, so that may not be so bad, but we'll see.
2017-03-02 19:29:28 -08:00
}
};
}
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
if let Some(start) = start_time_for_filename {
let tm = time::at(time::Timespec{sec: start.unix_seconds(), nsec: 0});
let stream_abbrev = if stream_type == db::StreamType::MAIN { "main" } else { "sub" };
let suffix = if mp4_type == mp4::Type::Normal { "mp4" } else { "m4s" };
builder.set_filename(&format!("{}-{}-{}.{}", tm.strftime("%Y%m%d%H%M%S").unwrap(),
camera_name, stream_abbrev, suffix))
.map_err(from_base_error)?;
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mp4 = builder.build(self.db.clone(), self.dirs_by_stream_id.clone())
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
.map_err(from_base_error)?;
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
if debug {
return Ok(plain_response(StatusCode::OK, format!("{:#?}", mp4)));
}
crate rename: http-(entity|file) -> http-serve
2018-01-23 11:08:21 -08:00
Ok(http_serve::serve(mp4, req))
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
fn static_file(&self, req: &Request<::hyper::Body>, path: &str) -> ResponseResult {
let s = self.ui_files.get(path).ok_or_else(|| not_found("no such static file"))?;
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
let f = tokio::task::block_in_place(move || {
fs::File::open(&s.path).map_err(internal_server_err)
})?;
update deps I want to use hyper::server::Request::bytes_mut(), so an update is needed. Update everything at once. Most notably, the http-serve update starts using the http crate types for some things. (More to come.)
2018-04-06 15:54:52 -07:00
let mut hdrs = http::HeaderMap::new();
upgrade to hyper 0.12.x Just one (intentional) functional change---now the streamers start shutting down while the webserver shuts down gracefully.
2018-08-29 22:26:19 -07:00
hdrs.insert(header::CONTENT_TYPE, s.mime.clone());
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
let e = http_serve::ChunkedReadFile::new(f, hdrs).map_err(internal_server_err)?;
crate rename: http-(entity|file) -> http-serve
2018-01-23 11:08:21 -08:00
Ok(http_serve::serve(e, &req))
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
fn authreq(&self, req: &Request<::hyper::Body>) -> auth::Request {
auth::Request {
when_sec: Some(self.db.clocks().realtime().sec),
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
addr: if self.trust_forward_hdrs {
req.headers().get("X-Real-IP")
.and_then(|v| v.to_str().ok())
.and_then(|v| IpAddr::from_str(v).ok())
} else { None },
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
user_agent: req.headers().get(header::USER_AGENT).map(|ua| ua.as_bytes().to_vec()),
}
}
endpoint to debug --trust-forward-hdrs for #26
2018-12-01 00:44:19 -08:00
fn request(&self, req: &Request<::hyper::Body>) -> ResponseResult {
let authreq = self.authreq(req);
document proxy setup in guide/secure.md (for #26) The guide is not as quick to follow and amateur-friendly as I'd like. A few things that might improve matters: * complete #27 (built-in https+letsencrypt), so that when not sharing the port, users don't need to use nginx or certbot. * more ubiquitous IPv6 (out of my control but should happen over time) to reduce need to share the port * embed a dynamic DNS client * support UPnP Internet Gateway Device Control Protocol (if common routers have this enabled? probably not for security reasons.) It's progress, though. Enough that I think I'll merge the auth branch into master shortly.
2018-12-27 16:00:15 -06:00
let host = req.headers().get(header::HOST).map(|h| String::from_utf8_lossy(h.as_bytes()));
let agent = authreq.user_agent.as_ref().map(|u| String::from_utf8_lossy(&u[..]));
endpoint to debug --trust-forward-hdrs for #26
2018-12-01 00:44:19 -08:00
Ok(plain_response(StatusCode::OK, format!(
"when: {}\n\
document proxy setup in guide/secure.md (for #26) The guide is not as quick to follow and amateur-friendly as I'd like. A few things that might improve matters: * complete #27 (built-in https+letsencrypt), so that when not sharing the port, users don't need to use nginx or certbot. * more ubiquitous IPv6 (out of my control but should happen over time) to reduce need to share the port * embed a dynamic DNS client * support UPnP Internet Gateway Device Control Protocol (if common routers have this enabled? probably not for security reasons.) It's progress, though. Enough that I think I'll merge the auth branch into master shortly.
2018-12-27 16:00:15 -06:00
host: {:?}\n\
endpoint to debug --trust-forward-hdrs for #26
2018-12-01 00:44:19 -08:00
addr: {:?}\n\
user_agent: {:?}\n\
secure: {:?}",
time::at(time::Timespec{sec: authreq.when_sec.unwrap(), nsec: 0})
.strftime("%FT%T")
.map(|f| f.to_string())
.unwrap_or_else(|e| e.to_string()),
document proxy setup in guide/secure.md (for #26) The guide is not as quick to follow and amateur-friendly as I'd like. A few things that might improve matters: * complete #27 (built-in https+letsencrypt), so that when not sharing the port, users don't need to use nginx or certbot. * more ubiquitous IPv6 (out of my control but should happen over time) to reduce need to share the port * embed a dynamic DNS client * support UPnP Internet Gateway Device Control Protocol (if common routers have this enabled? probably not for security reasons.) It's progress, though. Enough that I think I'll merge the auth branch into master shortly.
2018-12-27 16:00:15 -06:00
host.as_ref().map(|h| &*h),
endpoint to debug --trust-forward-hdrs for #26
2018-12-01 00:44:19 -08:00
&authreq.addr,
document proxy setup in guide/secure.md (for #26) The guide is not as quick to follow and amateur-friendly as I'd like. A few things that might improve matters: * complete #27 (built-in https+letsencrypt), so that when not sharing the port, users don't need to use nginx or certbot. * more ubiquitous IPv6 (out of my control but should happen over time) to reduce need to share the port * embed a dynamic DNS client * support UPnP Internet Gateway Device Control Protocol (if common routers have this enabled? probably not for security reasons.) It's progress, though. Enough that I think I'll merge the auth branch into master shortly.
2018-12-27 16:00:15 -06:00
agent.as_ref().map(|a| &*a),
endpoint to debug --trust-forward-hdrs for #26
2018-12-01 00:44:19 -08:00
self.is_secure(req))))
}
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
fn is_secure(&self, req: &Request<::hyper::Body>) -> bool {
self.trust_forward_hdrs &&
req.headers().get("X-Forwarded-Proto")
.map(|v| v.as_bytes() == b"https")
.unwrap_or(false)
}
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
fn login(&self, req: &Request<::hyper::Body>, body: Bytes) -> ResponseResult {
use application/json for login and logout
2020-01-08 23:23:58 -08:00
let r: json::LoginRequest = serde_json::from_slice(&body)
.map_err(|e| bad_req(e.to_string()))?;
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let authreq = self.authreq(req);
let host = req.headers().get(header::HOST).ok_or_else(|| bad_req("missing Host header!"))?;
let host = host.as_bytes();
let domain = match ::memchr::memchr(b':', host) {
Some(colon) => &host[0..colon],
None => host,
}.to_owned();
let mut l = self.db.lock();
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
let is_secure = self.is_secure(req);
let flags = (auth::SessionFlags::HttpOnly as i32) |
(auth::SessionFlags::SameSite as i32) |
use SameSite=Strict (for #26) I initially chose SameSite=Lax because I thought if a user followed a link to the landing page, the landing page's ajax requests wouldn't send the cookie. But I just did an experiment, and that's not true. Only the initial page load (of a .html file) lacks the cookie. All of its resources and ajax requests send the cookie. I'm not sure about document.cookie accesses, but my cookie is HttpOnly anyway, so it's irrelevant. So no reason to be lax.
2018-12-01 22:04:54 -08:00
(auth::SessionFlags::SameSiteStrict as i32) |
give downloaded .mp4s a useful filename This is effective both for Chrome's "Save As" dialog and for curl -OJ. It makes the filename like 20190717135519-driveway-main.mp4 rather than view.mp4 (Chrome) or view.mp4?s=33-36&ts=true (Curl).
2020-02-16 23:16:19 -08:00
if is_secure { auth::SessionFlags::Secure as i32 } else { 0 };
use application/json for login and logout
2020-01-08 23:23:58 -08:00
let (sid, _) = l.login_by_password(authreq, &r.username, r.password, Some(domain),
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
flags)
.map_err(|e| plain_response(StatusCode::UNAUTHORIZED, e.to_string()))?;
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
let s_suffix = if is_secure {
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
&b"; HttpOnly; Secure; SameSite=Strict; Max-Age=2147483648; Path=/"[..]
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
} else {
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
&b"; HttpOnly; SameSite=Strict; Max-Age=2147483648; Path=/"[..]
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
};
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mut encoded = [0u8; 64];
base64::encode_config_slice(&sid, base64::STANDARD_NO_PAD, &mut encoded);
let mut cookie = BytesMut::with_capacity("s=".len() + encoded.len() + s_suffix.len());
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
cookie.put(&b"s="[..]);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
cookie.put(&encoded[..]);
cookie.put(s_suffix);
Ok(Response::builder()
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
.header(header::SET_COOKIE, HeaderValue::from_maybe_shared(cookie.freeze())
.expect("cookie can't have invalid bytes"))
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.status(StatusCode::NO_CONTENT)
.body(b""[..].into()).unwrap())
}
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
fn logout(&self, req: &Request<hyper::Body>, body: Bytes) -> ResponseResult {
use application/json for login and logout
2020-01-08 23:23:58 -08:00
let r: json::LogoutRequest = serde_json::from_slice(&body)
.map_err(|e| bad_req(e.to_string()))?;
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mut res = Response::new(b""[..].into());
if let Some(sid) = extract_sid(req) {
let authreq = self.authreq(req);
let mut l = self.db.lock();
let hash = sid.hash();
let need_revoke = match l.authenticate_session(authreq.clone(), &hash) {
Ok((s, _)) => {
use application/json for login and logout
2020-01-08 23:23:58 -08:00
if !csrf_matches(r.csrf, s.csrf()) {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
warn!("logout request with missing/incorrect csrf");
return Err(bad_req("logout with incorrect csrf token"));
}
info!("revoking session");
true
},
Err(e) => {
// TODO: distinguish "no such session", "session is no longer valid", and
// "user ... is disabled" (which are all client error / bad state) from database
// errors.
warn!("logout failed: {}", e);
false
},
};
if need_revoke {
// TODO: inline this above with non-lexical lifetimes.
l.revoke_session(auth::RevocationReason::LoggedOut, None, authreq, &hash)
.map_err(internal_server_err)?;
}
// By now the session is invalid (whether it was valid to start with or not).
// Clear useless cookie.
res.headers_mut().append(header::SET_COOKIE,
HeaderValue::from_str("s=; Max-Age=0; Path=/").unwrap());
}
*res.status_mut() = StatusCode::NO_CONTENT;
Ok(res)
}
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
fn post_signals(&self, req: &Request<hyper::Body>, caller: Caller, body: Bytes)
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
-> ResponseResult {
if !caller.permissions.update_signals {
return Err(plain_response(StatusCode::UNAUTHORIZED, "update_signals required"));
}
web api glue for updating signals This is very lightly tested, but it at least sometimes works.
2019-06-14 16:11:12 -07:00
let r: json::PostSignalsRequest = serde_json::from_slice(&body)
.map_err(|e| bad_req(e.to_string()))?;
let mut l = self.db.lock();
let now = recording::Time::new(self.db.clocks().realtime());
let start = r.start_time_90k.map(recording::Time).unwrap_or(now);
let end = match r.end_base {
json::PostSignalsEndBase::Epoch => recording::Time(r.rel_end_time_90k.ok_or_else(
|| bad_req("must specify rel_end_time_90k when end_base is epoch"))?),
json::PostSignalsEndBase::Now => {
now + recording::Duration(r.rel_end_time_90k.unwrap_or(0))
},
};
l.update_signals(start .. end, &r.signal_ids, &r.states).map_err(from_base_error)?;
serve_json(req, &json::PostSignalsResponse {
time_90k: now.0,
})
}
fn get_signals(&self, req: &Request<hyper::Body>) -> ResponseResult {
db crate support for updating signals (#28) This is a definite work in progress. In particular, * there's no src/web.rs support yet so it can't be used, * the code is surprisingly complex, and there's almost no tests so far. I want to at least get complete branch coverage. * I may still go back to time_sec rather than time_90k to save RAM and flash. I simplified the approach a bit from the earlier goal in design/api.md. In particular, there's no longer the separate concept of "observation" vs "prediction". Now the predictions are just observations that extend a bit beyond now. They may be flushed prematurely and I'll try living with that to avoid making things even more complex.
2019-06-13 21:55:15 -07:00
let mut time = recording::Time::min_value() .. recording::Time::max_value();
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
if let Some(q) = req.uri().query() {
for (key, value) in form_urlencoded::parse(q.as_bytes()) {
let (key, value) = (key.borrow(), value.borrow());
match key {
"startTime90k" => {
time.start = recording::Time::parse(value)
.map_err(|_| bad_req("unparseable startTime90k"))?
},
"endTime90k" => {
time.end = recording::Time::parse(value)
.map_err(|_| bad_req("unparseable endTime90k"))?
},
_ => {},
}
}
}
let mut signals = json::Signals::default();
self.db.lock().list_changes_by_time(time, &mut |c: &db::signal::ListStateChangesRow| {
signals.times_90k.push(c.when.0);
signals.signal_ids.push(c.signal);
signals.states.push(c.state);
db crate support for updating signals (#28) This is a definite work in progress. In particular, * there's no src/web.rs support yet so it can't be used, * the code is surprisingly complex, and there's almost no tests so far. I want to at least get complete branch coverage. * I may still go back to time_sec rather than time_90k to save RAM and flash. I simplified the approach a bit from the earlier goal in design/api.md. In particular, there's no longer the separate concept of "observation" vs "prediction". Now the predictions are just observations that extend a bit beyond now. They may be flushed prematurely and I'll try living with that to avoid making things even more complex.
2019-06-13 21:55:15 -07:00
});
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
serve_json(req, &signals)
}
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
fn authenticate(&self, req: &Request<hyper::Body>, unauth_path: bool)
-> Result<Caller, base::Error> {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
if let Some(sid) = extract_sid(req) {
let authreq = self.authreq(req);
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
// TODO: real error handling! this assumes all errors are due to lack of
// authentication, when they could be logic errors in SQL or such.
if let Ok((s, u)) = self.db.lock().authenticate_session(authreq.clone(), &sid.hash()) {
return Ok(Caller {
permissions: s.permissions.clone(),
session: Some(json::Session {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
username: u.username.clone(),
csrf: s.csrf(),
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
}),
});
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
expose signal id in api ...and update api.md which described a different format than before or after.
2019-06-20 12:10:23 -07:00
info!("authenticate_session failed");
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
if let Some(s) = self.allow_unauthenticated_permissions.as_ref() {
return Ok(Caller {
permissions: s.clone(),
session: None,
});
}
if unauth_path {
return Ok(Caller {
permissions: db::Permissions::default(),
session: None,
})
}
bail_t!(Unauthenticated, "unauthenticated");
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
}
fn csrf_matches(csrf: &str, session: auth::SessionHash) -> bool {
let mut b64 = [0u8; 32];
session.encode_base64(&mut b64);
::ring::constant_time::verify_slices_are_equal(&b64[..], csrf.as_bytes()).is_ok()
}
/// Extracts `s` cookie from the HTTP request. Does not authenticate.
fn extract_sid(req: &Request<hyper::Body>) -> Option<auth::RawSessionId> {
let hdr = match req.headers().get(header::COOKIE) {
None => return None,
Some(c) => c,
};
for mut cookie in hdr.as_bytes().split(|&b| b == b';') {
if cookie.starts_with(b" ") {
cookie = &cookie[1..];
}
if cookie.starts_with(b"s=") {
let s = &cookie[2..];
if let Ok(s) = auth::RawSessionId::decode_base64(s) {
return Some(s);
}
}
}
None
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
}
use application/json for login and logout
2020-01-08 23:23:58 -08:00
/// Returns a future separating the request from its JSON body.
web api glue for updating signals This is very lightly tested, but it at least sometimes works.
2019-06-14 16:11:12 -07:00
///
/// If this is not a `POST` or the body's `Content-Type` is not
use application/json for login and logout
2020-01-08 23:23:58 -08:00
/// `application/json`, returns an appropriate error response instead.
web api glue for updating signals This is very lightly tested, but it at least sometimes works.
2019-06-14 16:11:12 -07:00
///
/// Use with `and_then` to chain logic which consumes the form body.
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
async fn with_json_body(mut req: Request<hyper::Body>)
-> Result<(Request<hyper::Body>, Bytes), Response<Body>> {
web api glue for updating signals This is very lightly tested, but it at least sometimes works.
2019-06-14 16:11:12 -07:00
if *req.method() != http::method::Method::POST {
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
return Err(plain_response(StatusCode::METHOD_NOT_ALLOWED, "POST expected"));
web api glue for updating signals This is very lightly tested, but it at least sometimes works.
2019-06-14 16:11:12 -07:00
}
let correct_mime_type = match req.headers().get(header::CONTENT_TYPE) {
Some(t) if t == "application/json" => true,
Some(t) if t == "application/json; charset=UTF-8" => true,
_ => false,
};
if !correct_mime_type {
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
return Err(bad_req("expected application/json request body"));
web api glue for updating signals This is very lightly tested, but it at least sometimes works.
2019-06-14 16:11:12 -07:00
}
let b = ::std::mem::replace(req.body_mut(), hyper::Body::empty());
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
match hyper::body::to_bytes(b).await {
Ok(b) => Ok((req, b)),
Err(e) => Err(internal_server_err(format_err!("unable to read request body: {}", e))),
}
web api glue for updating signals This is very lightly tested, but it at least sometimes works.
2019-06-14 16:11:12 -07:00
}
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
pub struct Config<'a> {
pub db: Arc<db::Database>,
pub ui_dir: Option<&'a str>,
pub trust_forward_hdrs: bool,
pub time_zone_name: String,
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
pub allow_unauthenticated_permissions: Option<db::Permissions>,
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
}
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
#[derive(Clone)]
pub struct Service(Arc<ServiceInner>);
impl Service {
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
pub fn new(config: Config) -> Result<Self, Error> {
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
let mut ui_files = HashMap::new();
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
if let Some(d) = config.ui_dir {
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
Service::fill_ui_files(d, &mut ui_files);
}
debug!("UI files: {:#?}", ui_files);
support multiple sample file directories This is still pretty basic support. There's no config UI support for renaming/moving the sample file directories after they are created, and no error checking that the files are still in the expected place. I can imagine sysadmins getting into trouble trying to change things. I hope to address at least some of that in a follow-up change to introduce a versioning/locking scheme that ensures databases and sample file dirs match in some way. A bonus change that kinda got pulled along for the ride: a dialog pops up in the config UI while a stream is being tested. The experience was pretty bad before; there was no indication the button worked at all until it was done, sometimes many seconds later.
2018-02-11 22:45:51 -08:00
let dirs_by_stream_id = {
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
let l = config.db.lock();
support multiple sample file directories This is still pretty basic support. There's no config UI support for renaming/moving the sample file directories after they are created, and no error checking that the files are still in the expected place. I can imagine sysadmins getting into trouble trying to change things. I hope to address at least some of that in a follow-up change to introduce a versioning/locking scheme that ensures databases and sample file dirs match in some way. A bonus change that kinda got pulled along for the ride: a dialog pops up in the config UI while a stream is being tested. The experience was pretty bad before; there was no indication the button worked at all until it was done, sometimes many seconds later.
2018-02-11 22:45:51 -08:00
let mut d =
FnvHashMap::with_capacity_and_hasher(l.streams_by_id().len(), Default::default());
for (&id, s) in l.streams_by_id().iter() {
let dir_id = match s.sample_file_dir_id {
Some(d) => d,
None => continue,
};
d.insert(id, l.sample_file_dirs_by_id()
.get(&dir_id)
.unwrap()
new database/sample file dir interlock scheme The idea is to avoid the problems described in src/schema.proto; those possibilities have bothered me for a while. A bonus is that (in a future commit) it can replace the sample file uuid scheme in favor of using <camera_uuid>-<stream_type>/<recording_id> for several advantages: * on data integrity problems (specifically, extra sample files), more information to use to understand what happened. * no more reserving sample files prior to using them. This avoids some extra database transactions on startup (now there's an extra two total rather than an extra one per stream). It also simplifies an upcoming change I want to make in which some streams are not flushed immediately, reducing the write load significantly (maybe one per minute total rather than one per stream per minute). * get rid of eight bytes per playback cache entry in RAM (and nine bytes per recording_playback row on flash). The implementation is still pretty rough in places: * Lack of tests. * Poor ode organization. In particular, SampleFileDirectory::write_meta shouldn't be exposed beyond db. I'm thinking about moving db.rs and SampleFileDirectory to a new crate, moonfire_nvr_db. This would improve compile times as well. * No tooling for renaming a sample file directory. * Config subcommand still panics in conditions that can be reasonably expected to happen.
2018-02-14 23:10:10 -08:00
.get()?);
support multiple sample file directories This is still pretty basic support. There's no config UI support for renaming/moving the sample file directories after they are created, and no error checking that the files are still in the expected place. I can imagine sysadmins getting into trouble trying to change things. I hope to address at least some of that in a follow-up change to introduce a versioning/locking scheme that ensures databases and sample file dirs match in some way. A bonus change that kinda got pulled along for the ride: a dialog pops up in the config UI while a stream is being tested. The experience was pretty bad before; there was no indication the button worked at all until it was done, sometimes many seconds later.
2018-02-11 22:45:51 -08:00
}
Arc::new(d)
};
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
Ok(Service(Arc::new(ServiceInner {
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
db: config.db,
support multiple sample file directories This is still pretty basic support. There's no config UI support for renaming/moving the sample file directories after they are created, and no error checking that the files are still in the expected place. I can imagine sysadmins getting into trouble trying to change things. I hope to address at least some of that in a follow-up change to introduce a versioning/locking scheme that ensures databases and sample file dirs match in some way. A bonus change that kinda got pulled along for the ride: a dialog pops up in the config UI while a stream is being tested. The experience was pretty bad before; there was no indication the button worked at all until it was done, sometimes many seconds later.
2018-02-11 22:45:51 -08:00
dirs_by_stream_id,
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
ui_files,
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
allow_unauthenticated_permissions: config.allow_unauthenticated_permissions,
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
trust_forward_hdrs: config.trust_forward_hdrs,
time_zone_name: config.time_zone_name,
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
})))
}
fn fill_ui_files(dir: &str, files: &mut HashMap<String, UiFile>) {
let r = match fs::read_dir(dir) {
Ok(r) => r,
Err(e) => {
warn!("Unable to search --ui-dir={}; will serve no static files. Error was: {}",
dir, e);
return;
}
};
for e in r {
let e = match e {
Ok(e) => e,
Err(e) => {
warn!("Error searching UI directory; may be missing files. Error was: {}", e);
continue;
},
};
let (p, mime) = match e.file_name().to_str() {
update deps I want to use hyper::server::Request::bytes_mut(), so an update is needed. Update everything at once. Most notably, the http-serve update starts using the http crate types for some things. (More to come.)
2018-04-06 15:54:52 -07:00
Some(n) if n == "index.html" => ("/".to_owned(), "text/html"),
Some(n) if n.ends_with(".html") => (format!("/{}", n), "text/html"),
Some(n) if n.ends_with(".ico") => (format!("/{}", n), "image/vnd.microsoft.icon"),
Some(n) if n.ends_with(".js") => (format!("/{}", n), "text/javascript"),
Some(n) if n.ends_with(".map") => (format!("/{}", n), "text/javascript"),
Some(n) if n.ends_with(".png") => (format!("/{}", n), "image/png"),
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
Some(n) => {
warn!("UI directory file {:?} has unknown extension; skipping", n);
continue;
},
None => {
warn!("UI directory file {:?} is not a valid UTF-8 string; skipping",
e.file_name());
continue;
},
};
files.insert(p, UiFile {
upgrade to hyper 0.12.x Just one (intentional) functional change---now the streamers start shutting down while the webserver shuts down gracefully.
2018-08-29 22:26:19 -07:00
mime: HeaderValue::from_static(mime),
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
path: e.path(),
});
}
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
switch to websocket for live stream (#59) The multipart stream / hanging GET approach worked in a prototype for a single stream, but Chrome has a per-host limit of six connections. If I try streaming all my cameras at once, I hit that limit. I can't open all the streams, much less additional connections to load init segments and such. Websockets apparently has a much higher limit of 256.
2020-02-28 21:41:31 -08:00
fn stream_live_m4s(&self, req: Request<::hyper::Body>, caller: Caller, uuid: Uuid,
backend support for live stream (#59) This is so far completely untested, for use by a new UI prototype. It creates a new URL endpoint which sends one video/mp4 media segment per key frame, with the dependent frames included. This means there will be about one key frame interval of latency (typically about a second). This seems hard to avoid, as mentioned in issue #59.
2019-01-21 15:58:52 -08:00
stream_type: db::StreamType) -> ResponseResult {
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
if !caller.permissions.view_video {
return Err(plain_response(StatusCode::UNAUTHORIZED, "view_video required"));
}
switch to websocket for live stream (#59) The multipart stream / hanging GET approach worked in a prototype for a single stream, but Chrome has a per-host limit of six connections. If I try streaming all my cameras at once, I hit that limit. I can't open all the streams, much less additional connections to load init segments and such. Websockets apparently has a much higher limit of 256.
2020-02-28 21:41:31 -08:00
backend support for live stream (#59) This is so far completely untested, for use by a new UI prototype. It creates a new URL endpoint which sends one video/mp4 media segment per key frame, with the dependent frames included. This means there will be about one key frame interval of latency (typically about a second). This seems hard to avoid, as mentioned in issue #59.
2019-01-21 15:58:52 -08:00
let stream_id;
let open_id;
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
let (sub_tx, sub_rx) = futures::channel::mpsc::unbounded();
backend support for live stream (#59) This is so far completely untested, for use by a new UI prototype. It creates a new URL endpoint which sends one video/mp4 media segment per key frame, with the dependent frames included. This means there will be about one key frame interval of latency (typically about a second). This seems hard to avoid, as mentioned in issue #59.
2019-01-21 15:58:52 -08:00
{
let mut db = self.0.db.lock();
open_id = match db.open {
None => return Err(plain_response(
StatusCode::PRECONDITION_FAILED,
"database is read-only; there are no live streams")),
Some(o) => o.id,
};
let camera = db.get_camera(uuid)
.ok_or_else(|| plain_response(StatusCode::NOT_FOUND,
format!("no such camera {}", uuid)))?;
stream_id = camera.streams[stream_type.index()]
.ok_or_else(|| plain_response(StatusCode::NOT_FOUND,
format!("no such stream {}/{}", uuid,
stream_type)))?;
db.watch_live(stream_id, Box::new(move |l| sub_tx.unbounded_send(l).is_ok()))
.expect("stream_id refed by camera");
}
switch to websocket for live stream (#59) The multipart stream / hanging GET approach worked in a prototype for a single stream, but Chrome has a per-host limit of six connections. If I try streaming all my cameras at once, I hit that limit. I can't open all the streams, much less additional connections to load init segments and such. Websockets apparently has a much higher limit of 256.
2020-02-28 21:41:31 -08:00
let (parts, body) = req.into_parts();
let req = Request::from_parts(parts, ());
let response = tungstenite::handshake::server::create_response(&req)
.map_err(|e| bad_req(e.to_string()))?;
let (parts, ()) = response.into_parts();
tokio::spawn(self.clone().stream_live_m4s_ws(stream_id, open_id, body, sub_rx));
Ok(Response::from_parts(parts, Body::from("")))
}
async fn stream_live_m4s_ws(
self, stream_id: i32, open_id: u32, body: hyper::Body,
mut sub_rx: futures::channel::mpsc::UnboundedReceiver<db::LiveSegment>) {
let upgraded = match body.on_upgrade().await {
Ok(u) => u,
Err(e) => {
warn!("Unable to upgrade stream to websocket: {}", e);
return;
},
};
let mut ws = tokio_tungstenite::WebSocketStream::from_raw_socket(
upgraded,
tungstenite::protocol::Role::Server,
None,
).await;
loop {
let live = match sub_rx.next().await {
Some(l) => l,
None => return,
};
if let Err(e) = self.stream_live_m4s_chunk(open_id, stream_id, &mut ws, live).await {
info!("Dropping WebSocket after error: {}", e);
return;
}
}
}
async fn stream_live_m4s_chunk(
&self, open_id: u32, stream_id: i32,
ws: &mut tokio_tungstenite::WebSocketStream<hyper::upgrade::Upgraded>,
live: db::LiveSegment) -> Result<(), Error> {
let mut builder = mp4::FileBuilder::new(mp4::Type::MediaSegment);
let mut vse_id = None;
let mut start = None;
{
let db = self.0.db.lock();
let mut rows = 0;
db.list_recordings_by_id(stream_id, live.recording .. live.recording+1, &mut |r| {
rows += 1;
let vse = db.video_sample_entries_by_id().get(&r.video_sample_entry_id)
.unwrap();
vse_id = Some(strutil::hex(&vse.sha1));
start = Some(r.start);
builder.append(&db, r, live.off_90k.clone())?;
Ok(())
})?;
if rows != 1 {
bail_t!(Internal, "unable to find {:?}", live);
}
}
let vse_id = vse_id.unwrap();
let start = start.unwrap();
use http_serve::Entity;
let mp4 = builder.build(self.0.db.clone(), self.0.dirs_by_stream_id.clone())?;
let mut hdrs = http::header::HeaderMap::new();
mp4.add_headers(&mut hdrs);
let mime_type = hdrs.get(http::header::CONTENT_TYPE).unwrap();
let hdr = format!(
"Content-Type: {}\r\n\
X-Recording-Start: {}\r\n\
X-Recording-Id: {}.{}\r\n\
X-Time-Range: {}-{}\r\n\
X-Video-Sample-Entry-Sha1: {}\r\n\r\n",
mime_type.to_str().unwrap(),
start.0,
open_id,
live.recording,
live.off_90k.start,
live.off_90k.end,
&vse_id);
let mut v = /*Pin::from(*/hdr.into_bytes()/*)*/;
mp4.append_into_vec(&mut v).await?;
//let v = Pin::into_inner();
ws.send(tungstenite::Message::Binary(v)).await?;
Ok(())
backend support for live stream (#59) This is so far completely untested, for use by a new UI prototype. It creates a new URL endpoint which sends one video/mp4 media segment per key frame, with the dependent frames included. This means there will be about one key frame interval of latency (typically about a second). This seems hard to avoid, as mentioned in issue #59.
2019-01-21 15:58:52 -08:00
}
web api glue for updating signals This is very lightly tested, but it at least sometimes works.
2019-06-14 16:11:12 -07:00
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
fn signals(&self, req: Request<hyper::Body>, caller: Caller)
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
-> Box<dyn Future<Output = Result<Response<Body>, Response<Body>>> + Send + Sync + 'static> {
web api glue for updating signals This is very lightly tested, but it at least sometimes works.
2019-06-14 16:11:12 -07:00
use http::method::Method;
match *req.method() {
Method::POST => Box::new(with_json_body(req)
.and_then({
let s = self.0.clone();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
move |(req, b)| future::ready(s.post_signals(&req, caller, b))
web api glue for updating signals This is very lightly tested, but it at least sometimes works.
2019-06-14 16:11:12 -07:00
})),
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
Method::GET | Method::HEAD => Box::new(future::ready(self.0.get_signals(&req))),
web api glue for updating signals This is very lightly tested, but it at least sometimes works.
2019-06-14 16:11:12 -07:00
_ => Box::new(future::err(plain_response(StatusCode::METHOD_NOT_ALLOWED,
"POST, GET, or HEAD expected"))),
}
}
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
pub fn serve(&mut self, req: Request<::hyper::Body>) -> BoxedFuture {
fn wrap<R>(is_private: bool, r: R) -> BoxedFuture
where R: Future<Output = Result<Response<Body>, Response<Body>>> + Send + Sync + 'static {
return Box::new(r.or_else(|e| futures::future::ok(e)).map_ok(move |mut r| {
add Cache-Control: private hdr to non-static stuff for #26
2018-12-01 16:55:10 -08:00
if is_private {
r.headers_mut().insert("Cache-Control", HeaderValue::from_static("private"));
}
r
}))
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
add Cache-Control: private hdr to non-static stuff for #26
2018-12-01 16:55:10 -08:00
fn wrap_r(is_private: bool, r: ResponseResult)
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
-> Box<dyn Future<Output = Result<Response<Body>, BoxedError>> + Send + Sync + 'static> {
return wrap(is_private, future::ready(r))
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
let p = Path::decode(req.uri().path());
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
let always_allow_unauthenticated = match p {
Path::NotFound | Path::Request | Path::Login | Path::Logout | Path::Static => true,
_ => false,
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
};
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
debug!("request on: {}: {:?}", req.uri(), p);
let caller = match self.0.authenticate(&req, always_allow_unauthenticated) {
Ok(c) => c,
Err(e) => return Box::new(future::ok(from_base_error(e))),
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
};
add Cache-Control: private hdr to non-static stuff for #26
2018-12-01 16:55:10 -08:00
match p {
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
Path::InitSegment(sha1, debug) => wrap_r(true, self.0.init_segment(sha1, debug, &req)),
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
Path::TopLevel => wrap_r(true, self.0.top_level(&req, caller)),
add Cache-Control: private hdr to non-static stuff for #26
2018-12-01 16:55:10 -08:00
Path::Request => wrap_r(true, self.0.request(&req)),
Path::Camera(uuid) => wrap_r(true, self.0.camera(&req, uuid)),
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
Path::StreamRecordings(uuid, type_) => {
add Cache-Control: private hdr to non-static stuff for #26
2018-12-01 16:55:10 -08:00
wrap_r(true, self.0.stream_recordings(&req, uuid, type_))
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
},
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
Path::StreamViewMp4(uuid, type_, debug) => {
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
wrap_r(true, self.0.stream_view_mp4(&req, caller, uuid, type_, mp4::Type::Normal,
debug))
support segmented mp4s This is intended to support HTML5 Media Source Extensions, which I expect to be the most practical way to make a good web UI with a proper scrub bar and such. This feature has had very limited testing on Chrome and Firefox, and that was not entirely successful. More work is needed before it's usable, but this seems like a helpful progress checkpoint.
2017-10-01 15:29:22 -07:00
},
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
Path::StreamViewMp4Segment(uuid, type_, debug) => {
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
wrap_r(true, self.0.stream_view_mp4(&req, caller, uuid, type_,
mp4::Type::MediaSegment, debug))
support segmented mp4s This is intended to support HTML5 Media Source Extensions, which I expect to be the most practical way to make a good web UI with a proper scrub bar and such. This feature has had very limited testing on Chrome and Firefox, and that was not entirely successful. More work is needed before it's usable, but this seems like a helpful progress checkpoint.
2017-10-01 15:29:22 -07:00
},
backend support for live stream (#59) This is so far completely untested, for use by a new UI prototype. It creates a new URL endpoint which sends one video/mp4 media segment per key frame, with the dependent frames included. This means there will be about one key frame interval of latency (typically about a second). This seems hard to avoid, as mentioned in issue #59.
2019-01-21 15:58:52 -08:00
Path::StreamLiveMp4Segments(uuid, type_) => {
switch to websocket for live stream (#59) The multipart stream / hanging GET approach worked in a prototype for a single stream, but Chrome has a per-host limit of six connections. If I try streaming all my cameras at once, I hit that limit. I can't open all the streams, much less additional connections to load init segments and such. Websockets apparently has a much higher limit of 256.
2020-02-28 21:41:31 -08:00
wrap_r(true, self.stream_live_m4s(req, caller, uuid, type_))
backend support for live stream (#59) This is so far completely untested, for use by a new UI prototype. It creates a new URL endpoint which sends one video/mp4 media segment per key frame, with the dependent frames included. This means there will be about one key frame interval of latency (typically about a second). This seems hard to avoid, as mentioned in issue #59.
2019-01-21 15:58:52 -08:00
},
add Cache-Control: private hdr to non-static stuff for #26
2018-12-01 16:55:10 -08:00
Path::NotFound => wrap(true, future::err(not_found("path not understood"))),
use application/json for login and logout
2020-01-08 23:23:58 -08:00
Path::Login => wrap(true, with_json_body(req).and_then({
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let s = self.clone();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
move |(req, b)| future::ready(s.0.login(&req, b))
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
})),
use application/json for login and logout
2020-01-08 23:23:58 -08:00
Path::Logout => wrap(true, with_json_body(req).and_then({
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let s = self.clone();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
move |(req, b)| future::ready(s.0.logout(&req, b))
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
})),
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
Path::Signals => wrap(true, Pin::from(self.signals(req, caller))),
add Cache-Control: private hdr to non-static stuff for #26
2018-12-01 16:55:10 -08:00
Path::Static => wrap_r(false, self.0.static_file(&req, req.uri().path())),
upgrade to hyper 0.12.x Just one (intentional) functional change---now the streamers start shutting down while the webserver shuts down gracefully.
2018-08-29 22:26:19 -07:00
}
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
}
#[cfg(test)]
mod tests {
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
use db::testutil::{self, TestDb};
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
use futures::future::FutureExt;
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
use log::info;
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
use std::collections::HashMap;
add a basic Javascript UI The Javascript is pretty amateurish I'm sure but at least it's something to iterate from. It's already much more pleasant for browsing through videos in several ways: * more responsive to load only a day at a time rather than 90+ days * much easier to see the same time segment on several cameras * more pleasant to have the videos load as a popup rather than a link that blows away your position in an enormous list * exposes the fancier .mp4 generation options: splitting at lengths other than the default, trimming to an arbitrary start and end time, including a subtitle track with timestamps. There's a slight regression in functionality: I didn't match the former top-level page which showed how much camera used of its disk allocation and the total duration of video. This is exposed in the JSON API, so it shouldn't be too hard to add back.
2017-10-21 21:54:27 -07:00
use super::Segments;
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
struct Server {
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
db: TestDb<base::clock::RealClocks>,
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
base_url: String,
//test_camera_uuid: Uuid,
handle: Option<::std::thread::JoinHandle<()>>,
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
shutdown_tx: Option<futures::channel::oneshot::Sender<()>>,
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
impl Server {
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
fn new(allow_unauthenticated_permissions: Option<db::Permissions>) -> Server {
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
let db = TestDb::new(base::clock::RealClocks {});
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
let (shutdown_tx, shutdown_rx) = futures::channel::oneshot::channel::<()>();
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
let service = super::Service::new(super::Config {
db: db.db.clone(),
ui_dir: None,
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
allow_unauthenticated_permissions,
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
trust_forward_hdrs: true,
time_zone_name: "".to_owned(),
}).unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
let make_svc = hyper::service::make_service_fn(move |_conn| {
futures::future::ok::<_, std::convert::Infallible>(hyper::service::service_fn({
let mut s = service.clone();
move |req| std::pin::Pin::from(s.serve(req))
}))
});
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let (tx, rx) = std::sync::mpsc::channel();
let handle = ::std::thread::spawn(move || {
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
let addr = ([127, 0, 0, 1], 0).into();
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let mut rt = tokio::runtime::Runtime::new().unwrap();
let srv = rt.enter(|| {
hyper::server::Server::bind(&addr)
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
.tcp_nodelay(true)
.serve(make_svc)
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
});
let addr = srv.local_addr(); // resolve port 0 to a real ephemeral port number.
tx.send(addr).unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
rt.block_on(srv.with_graceful_shutdown(shutdown_rx.map(|_| ()))).unwrap();
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
});
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let addr = rx.recv().unwrap();
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
// Create a user.
let mut c = db::UserChange::add_user("slamb".to_owned());
c.set_password("hunter2".to_owned());
db.db.lock().apply_user_change(c).unwrap();
Server {
db,
base_url: format!("http://{}:{}", addr.ip(), addr.port()),
handle: Some(handle),
shutdown_tx: Some(shutdown_tx),
}
}
}
impl Drop for Server {
fn drop(&mut self) {
self.shutdown_tx.take().unwrap().send(()).unwrap();
self.handle.take().unwrap().join().unwrap()
}
}
#[derive(Clone, Debug, Default)]
struct SessionCookie(Option<String>);
impl SessionCookie {
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
pub fn new(headers: &reqwest::header::HeaderMap) -> Self {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mut c = SessionCookie::default();
c.update(headers);
c
}
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
pub fn update(&mut self, headers: &reqwest::header::HeaderMap) {
for set_cookie in headers.get_all(reqwest::header::SET_COOKIE) {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mut set_cookie = set_cookie.to_str().unwrap().split("; ");
let c = set_cookie.next().unwrap();
let mut clear = false;
for attr in set_cookie {
if attr == "Max-Age=0" {
clear = true;
}
}
if !c.starts_with("s=") {
panic!("unrecognized cookie");
}
self.0 = if clear { None } else { Some(c.to_owned()) };
}
}
/// Produces a `Cookie` header value.
pub fn header(&self) -> String {
self.0.as_ref().map(|s| s.as_str()).unwrap_or("").to_owned()
}
}
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
#[test]
fn paths() {
use super::Path;
use uuid::Uuid;
let cam_uuid = Uuid::parse_str("35144640-ff1e-4619-b0d5-4c74c185741c").unwrap();
assert_eq!(Path::decode("/foo"), Path::Static);
assert_eq!(Path::decode("/api/"), Path::TopLevel);
assert_eq!(Path::decode("/api/init/07cec464126825088ea86a07eddd6a00afa71559.mp4"),
Path::InitSegment([0x07, 0xce, 0xc4, 0x64, 0x12, 0x68, 0x25, 0x08, 0x8e, 0xa8,
0x6a, 0x07, 0xed, 0xdd, 0x6a, 0x00, 0xaf, 0xa7, 0x15, 0x59],
false));
assert_eq!(Path::decode("/api/init/07cec464126825088ea86a07eddd6a00afa71559.mp4.txt"),
Path::InitSegment([0x07, 0xce, 0xc4, 0x64, 0x12, 0x68, 0x25, 0x08, 0x8e, 0xa8,
0x6a, 0x07, 0xed, 0xdd, 0x6a, 0x00, 0xaf, 0xa7, 0x15, 0x59],
true));
assert_eq!(Path::decode("/api/init/000000000000000000000000000000000000000x.mp4"),
Path::NotFound); // non-hexadigit
assert_eq!(Path::decode("/api/init/000000000000000000000000000000000000000.mp4"),
Path::NotFound); // too short
assert_eq!(Path::decode("/api/cameras/35144640-ff1e-4619-b0d5-4c74c185741c/"),
Path::Camera(cam_uuid));
assert_eq!(Path::decode("/api/cameras/asdf/"), Path::NotFound);
assert_eq!(
Path::decode("/api/cameras/35144640-ff1e-4619-b0d5-4c74c185741c/main/recordings"),
Path::StreamRecordings(cam_uuid, db::StreamType::MAIN));
assert_eq!(
Path::decode("/api/cameras/35144640-ff1e-4619-b0d5-4c74c185741c/sub/recordings"),
Path::StreamRecordings(cam_uuid, db::StreamType::SUB));
assert_eq!(
Path::decode("/api/cameras/35144640-ff1e-4619-b0d5-4c74c185741c/junk/recordings"),
Path::NotFound);
assert_eq!(
Path::decode("/api/cameras/35144640-ff1e-4619-b0d5-4c74c185741c/main/view.mp4"),
Path::StreamViewMp4(cam_uuid, db::StreamType::MAIN, false));
assert_eq!(
Path::decode("/api/cameras/35144640-ff1e-4619-b0d5-4c74c185741c/main/view.mp4.txt"),
Path::StreamViewMp4(cam_uuid, db::StreamType::MAIN, true));
assert_eq!(
Path::decode("/api/cameras/35144640-ff1e-4619-b0d5-4c74c185741c/main/view.m4s"),
Path::StreamViewMp4Segment(cam_uuid, db::StreamType::MAIN, false));
assert_eq!(
Path::decode("/api/cameras/35144640-ff1e-4619-b0d5-4c74c185741c/main/view.m4s.txt"),
Path::StreamViewMp4Segment(cam_uuid, db::StreamType::MAIN, true));
backend support for live stream (#59) This is so far completely untested, for use by a new UI prototype. It creates a new URL endpoint which sends one video/mp4 media segment per key frame, with the dependent frames included. This means there will be about one key frame interval of latency (typically about a second). This seems hard to avoid, as mentioned in issue #59.
2019-01-21 15:58:52 -08:00
assert_eq!(
Path::decode("/api/cameras/35144640-ff1e-4619-b0d5-4c74c185741c/main/live.m4s"),
Path::StreamLiveMp4Segments(cam_uuid, db::StreamType::MAIN));
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
assert_eq!(
Path::decode("/api/cameras/35144640-ff1e-4619-b0d5-4c74c185741c/main/junk"),
Path::NotFound);
assert_eq!(Path::decode("/api/login"), Path::Login);
assert_eq!(Path::decode("/api/logout"), Path::Logout);
read-only signals support (#28) This is mostly untested and useless by itself, but it's a starting point. In particular: * there's no way to set up signals or add/remove/update events yet except by manual changes to the database. * if you associate a signal with a camera then remove the camera, hitting /api/ will error out.
2019-06-06 16:18:13 -07:00
assert_eq!(Path::decode("/api/signals"), Path::Signals);
add a url for getting debug info about a .mp4 file and add a unit test of path decoding along the way
2018-12-29 13:06:44 -06:00
assert_eq!(Path::decode("/api/junk"), Path::NotFound);
}
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
#[test]
fn test_segments() {
testutil::init();
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
assert_eq!(Segments{ids: 1..2, open_id: None, start_time: 0, end_time: None},
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
Segments::parse("1").unwrap());
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
assert_eq!(Segments{ids: 1..2, open_id: Some(42), start_time: 0, end_time: None},
Segments::parse("1@42").unwrap());
assert_eq!(Segments{ids: 1..2, open_id: None, start_time: 26, end_time: None},
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
Segments::parse("1.26-").unwrap());
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
assert_eq!(Segments{ids: 1..2, open_id: Some(42), start_time: 26, end_time: None},
Segments::parse("1@42.26-").unwrap());
assert_eq!(Segments{ids: 1..2, open_id: None, start_time: 0, end_time: Some(42)},
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
Segments::parse("1.-42").unwrap());
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
assert_eq!(Segments{ids: 1..2, open_id: None, start_time: 26, end_time: Some(42)},
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
Segments::parse("1.26-42").unwrap());
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
assert_eq!(Segments{ids: 1..6, open_id: None, start_time: 0, end_time: None},
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
Segments::parse("1-5").unwrap());
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
assert_eq!(Segments{ids: 1..6, open_id: None, start_time: 26, end_time: None},
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
Segments::parse("1-5.26-").unwrap());
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
assert_eq!(Segments{ids: 1..6, open_id: None, start_time: 0, end_time: Some(42)},
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
Segments::parse("1-5.-42").unwrap());
allow listing and viewing uncommitted recordings There may be considerable lag between being fully written and being committed when using the flush_if_sec feature. Additionally, this is a step toward listing and viewing recordings before they're fully written. That's a considerable delay: 60 to 120 seconds for the first recording of a run, 0 to 60 seconds for subsequent recordings. These recordings aren't yet included in the information returned by /api/?days=true. They probably should be, but small steps.
2018-03-02 11:38:11 -08:00
assert_eq!(Segments{ids: 1..6, open_id: None, start_time: 26, end_time: Some(42)},
schema version 1 The advantages of the new schema are: * overlapping recordings can be unambiguously described and viewed. This is a significant problem right now; the clock on my cameras appears to run faster than the (NTP-synchronized) clock on my NVR. Thus, if an RTSP session drops and is quickly reconnected, there's likely to be overlap. * less I/O is required to view mp4s when there are multiple cameras. This is a pretty dramatic difference in the number of database read syscalls with pragma page_size = 1024 (605 -> 39 in one test), although I'm not sure how much of that maps to actual I/O wait time. That's probably as dramatic as it is due to overflow page chaining. But even with larger page sizes, there's an improvement. It helps to stop interleaving the video_index fields from different cameras. There are changes to the JSON API to take advantage of this, described in design/api.md. There's an upgrade procedure, described in guide/schema.md.
2016-12-20 22:08:18 -08:00
Segments::parse("1-5.26-42").unwrap());
}
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
#[tokio::test]
async fn unauthorized_without_cookie() {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
testutil::init();
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
let s = Server::new(None);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let cli = reqwest::Client::new();
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let resp = cli.get(&format!("{}/api/", &s.base_url)).send().await.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::UNAUTHORIZED);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
#[tokio::test]
async fn login() {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
testutil::init();
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
let s = Server::new(None);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let cli = reqwest::Client::new();
let login_url = format!("{}/api/login", &s.base_url);
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let resp = cli.get(&login_url).send().await.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::METHOD_NOT_ALLOWED);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let resp = cli.post(&login_url).send().await.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::BAD_REQUEST);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mut p = HashMap::new();
p.insert("username", "slamb");
p.insert("password", "asdf");
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let resp = cli.post(&login_url).json(&p).send().await.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::UNAUTHORIZED);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
p.insert("password", "hunter2");
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let resp = cli.post(&login_url).json(&p).send().await.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::NO_CONTENT);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let cookie = SessionCookie::new(resp.headers());
info!("cookie: {:?}", cookie);
info!("header: {}", cookie.header());
let resp = cli.get(&format!("{}/api/", &s.base_url))
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
.header(reqwest::header::COOKIE, cookie.header())
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.send()
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
.await
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::OK);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
#[tokio::test]
async fn logout() {
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
testutil::init();
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
let s = Server::new(None);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let cli = reqwest::Client::new();
let mut p = HashMap::new();
p.insert("username", "slamb");
p.insert("password", "hunter2");
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
let resp = cli.post(&format!("{}/api/login", &s.base_url)).json(&p).send().await.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::NO_CONTENT);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let cookie = SessionCookie::new(resp.headers());
// A GET shouldn't work.
let resp = cli.get(&format!("{}/api/logout", &s.base_url))
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
.header(reqwest::header::COOKIE, cookie.header())
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.send()
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
.await
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::METHOD_NOT_ALLOWED);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
// Neither should a POST without a csrf token.
let resp = cli.post(&format!("{}/api/logout", &s.base_url))
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
.header(reqwest::header::COOKIE, cookie.header())
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.send()
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
.await
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::BAD_REQUEST);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
// But it should work with the csrf token.
// Retrieve that from the toplevel API request.
let toplevel: serde_json::Value = cli.post(&format!("{}/api/", &s.base_url))
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
.header(reqwest::header::COOKIE, cookie.header())
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
.send().await.unwrap()
.json().await.unwrap();
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let csrf = toplevel.get("session").unwrap().get("csrf").unwrap().as_str();
let mut p = HashMap::new();
p.insert("csrf", csrf);
let resp = cli.post(&format!("{}/api/logout", &s.base_url))
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
.header(reqwest::header::COOKIE, cookie.header())
use application/json for login and logout
2020-01-08 23:23:58 -08:00
.json(&p)
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.send()
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
.await
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::NO_CONTENT);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
let mut updated_cookie = cookie.clone();
updated_cookie.update(resp.headers());
// The cookie should be cleared client-side.
assert!(updated_cookie.0.is_none());
// It should also be invalidated server-side.
let resp = cli.get(&format!("{}/api/", &s.base_url))
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
.header(reqwest::header::COOKIE, cookie.header())
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.send()
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
.await
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::UNAUTHORIZED);
preliminary web support for auth (#26) Some caveats: * it doesn't record the peer IP yet, which makes it harder to verify sessions are valid. This is a little annoying to do in hyper now (see hyperium/hyper#1410). The direct peer might not be what we want right now anyway because there's no TLS support yet (see #27). In the meantime, the sane way to expose Moonfire NVR to the Internet is via a proxy server, and recording the proxy's IP is not useful. Maybe better to interpret a RFC 7239 Forwarded header (and/or the older X-Forwarded-{For,Proto} headers). * it doesn't ever use Secure (https-only) cookies, for a similar reason. It's not safe to use even with a tls proxy until this is fixed. * there's no "moonfire-nvr config" support for inspecting/invalidating sessions yet. * in debug builds, logging in is crazy slow. See libpasta/libpasta#9. Some notes: * I removed the Javascript "no-use-before-defined" lint, as some of the functions form a cycle. * Fixed #20 along the way. I needed to add support for properly returning non-OK HTTP statuses to signal unauthorized and such. * I removed the Access-Control-Allow-Origin header support, which was at odds with the "SameSite=lax" in the cookie header. The "yarn start" method for running a local proxy server accomplishes the same thing as the Access-Control-Allow-Origin support in a more secure manner.
2018-11-25 21:31:50 -08:00
}
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
#[tokio::test]
async fn view_without_segments() {
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
testutil::init();
add concept of user/session permissions (I also considered the names "capabilities" and "scopes", but I think "permissions" is the most widely understood.) This is increasingly necessary as the web API becomes more capable. Among other things, it allows: * non-administrator users who can view but not access camera passwords or change any state * workers that update signal state based on cameras' built-in motion detection or a security system's events but don't need to view videos * control over what can be done without authenticating Currently session permissions are just copied from user permissions, but you can also imagine admin sessions vs not, as a checkbox when signing in. This would match the standard Unix workflow of using a non-administrative session most of the time. Relevant to my current signals work (#28) and to the addition of an administrative API (#35, including #66).
2019-06-19 15:17:50 -07:00
let mut permissions = db::Permissions::new();
permissions.view_video = true;
let s = Server::new(Some(permissions));
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
let cli = reqwest::Client::new();
let resp = cli.get(
&format!("{}/api/cameras/{}/main/view.mp4", &s.base_url, s.db.test_camera_uuid))
upgrade some deps, including reqwest The reqwest one is particularly notable because it means not having two versions of hyper/http/tokio/futures/bytes. It also drops a number of transitive deps; with some work I think I could stop depending on regex now.
2020-01-09 20:06:30 -08:00
.send().await.unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
assert_eq!(resp.status(), reqwest::StatusCode::BAD_REQUEST);
introduce typed errors and use in mp4 code Fixes #46. If there are no video_sample_entries, it returns InvalidArgument, which gets mapped to a HTTP 400. Various other failures turn into non-500s as well. There are many places that can & should be using typed errors, but it's a start.
2018-12-28 17:30:33 -06:00
}
Rust rewrite I should have submitted/pushed more incrementally but just played with it on my computer as I was learning the language. The new Rust version more or less matches the functionality of the current C++ version, although there are many caveats listed below. Upgrade notes: when moving from the C++ version, I recommend dropping and recreating the "recording_cover" index in SQLite3 to pick up the addition of the "video_sync_samples" column: $ sudo systemctl stop moonfire-nvr $ sudo -u moonfire-nvr sqlite3 /var/lib/moonfire-nvr/db/db sqlite> drop index recording_cover; sqlite3> create index ...rest of command as in schema.sql...; sqlite3> ^D Some known visible differences from the C++ version: * .mp4 generation queries SQLite3 differently. Before it would just get all video indexes in a single query. Now it leads with a query that should be satisfiable by the covering index (assuming the index has been recreated as noted above), then queries individual recording's indexes as needed to fill a LRU cache. I believe this is roughly similar speed for the initial hit (which generates the moov part of the file) and significantly faster when seeking. I would have done it a while ago with the C++ version but didn't want to track down a lru cache library. It was easier to find with Rust. * On startup, the Rust version cleans up old reserved files. This is as in the design; the C++ version was just missing this code. * The .html recording list output is a little different. It's in ascending order, with the most current segment shorten than an hour rather than the oldest. This is less ergonomic, but it was easy. I could fix it or just wait to obsolete it with some fancier JavaScript UI. * commandline argument parsing and logging have changed formats due to different underlying libraries. * The JSON output isn't quite right (matching the spec / C++ implementation) yet. Additional caveats: * I haven't done any proof-reading of prep.sh + install instructions. * There's a lot of code quality work to do: adding (back) comments and test coverage, developing a good Rust style. * The ffmpeg foreign function interface is particularly sketchy. I'd eventually like to switch to something based on autogenerated bindings. I'd also like to use pure Rust code where practical, but once I do on-NVR motion detection I'll need to existing C/C++ libraries for speed (H.264 decoding + OpenCL-based analysis).
2016-11-25 14:34:00 -08:00
}
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
#[cfg(all(test, feature="nightly"))]
mod bench {
extern crate test;
initial split of database to a separate crate It should reduce compile time / memory usage to put quite a bit of the code into a separate crate. I also intend to limit visibility of some things to only within the db crate, but that's for a future change. This is the smallest move that will compile.
2018-02-20 23:15:39 -08:00
use db::testutil::{self, TestDb};
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
use hyper;
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
use lazy_static::lazy_static;
use add_camera in tests, not direct db inserts This is a wash in terms of lines of code now, but it makes it a bit easier to maintain as I make changes to the schema (such as separating out streams from cameras), and it helps ensure the tests reflect reality.
2018-02-03 21:56:04 -08:00
use uuid::Uuid;
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
struct Server {
base_url: String,
use add_camera in tests, not direct db inserts This is a wash in terms of lines of code now, but it makes it a bit easier to maintain as I make changes to the schema (such as separating out streams from cameras), and it helps ensure the tests reflect reality.
2018-02-03 21:56:04 -08:00
test_camera_uuid: Uuid,
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
}
impl Server {
fn new() -> Server {
refine flush_if_sec behavior The new behavior eliminates a couple unpleasant edge cases in which it would never flush: * if all recording stops, whatever was unflushed would stay that way * if every recording attempt produces a 0-duration recording (such as if the camera sends only one frame and thus no PTS delta can be calculated), the list of recordings to flush would continue to grow
2018-03-23 15:16:43 -07:00
let db = TestDb::new(::base::clock::RealClocks {});
use add_camera in tests, not direct db inserts This is a wash in terms of lines of code now, but it makes it a bit easier to maintain as I make changes to the schema (such as separating out streams from cameras), and it helps ensure the tests reflect reality.
2018-02-03 21:56:04 -08:00
let test_camera_uuid = db.test_camera_uuid;
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
testutil::add_dummy_recordings_to_db(&db.db, 1440);
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
let service = super::Service::new(super::Config {
db: db.db.clone(),
ui_dir: None,
unbreak test --features=nightly
2019-06-20 14:57:13 -07:00
allow_unauthenticated_permissions: Some(db::Permissions::default()),
support proxy forwarded headers I went with legacy headers (X-Real-IP, X-Forwarded-Proto) because they appear to be more widely supported than the RFC 7239 Forwarded header.
2018-11-28 14:22:30 -08:00
trust_forward_hdrs: false,
time_zone_name: "".to_owned(),
}).unwrap();
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
let make_svc = hyper::service::make_service_fn(move |_conn| {
futures::future::ok::<_, std::convert::Infallible>(hyper::service::service_fn({
let mut s = service.clone();
move |req| std::pin::Pin::from(s.serve(req))
}))
});
let mut rt = tokio::runtime::Runtime::new().unwrap();
let srv = rt.enter(|| {
let addr = ([127, 0, 0, 1], 0).into();
hyper::server::Server::bind(&addr)
.tcp_nodelay(true)
.serve(make_svc)
});
let addr = srv.local_addr(); // resolve port 0 to a real ephemeral port number.
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
::std::thread::spawn(move || {
upgrade to hyper 0.13 ecosystem This doesn't take much advantage of async fns so far. For example, the with_{form,json}_body functions are still designed to be used with future combinators when it'd be more natural to call them from async fns now. But it's a start. Similarly, this still uses the old version of reqwest. Small steps. Requires Rust 1.40 now. (1.39 is a requirement of async, and 1.40 is a requirement of http-serve 0.2.0.)
2020-01-08 23:04:36 -08:00
rt.block_on(srv).unwrap();
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
});
use add_camera in tests, not direct db inserts This is a wash in terms of lines of code now, but it makes it a bit easier to maintain as I make changes to the schema (such as separating out streams from cameras), and it helps ensure the tests reflect reality.
2018-02-03 21:56:04 -08:00
Server {
base_url: format!("http://{}:{}", addr.ip(), addr.port()),
test_camera_uuid,
}
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
}
}
lazy_static! {
static ref SERVER: Server = { Server::new() };
}
#[bench]
lose "extern crate" everywhere (Rust 2018 edition)
2018-12-28 21:53:29 -06:00
fn serve_stream_recordings(b: &mut test::Bencher) {
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
testutil::init();
let server = &*SERVER;
schema version 2: support sub streams This allows each camera to have a main and a sub stream. Previously there was a field in the schema for the sub stream's url, but it didn't do anything. Now you can configure individual retention for main and sub streams. They show up grouped in the UI. No support for upgrading from schema version 1 yet.
2018-01-23 11:05:07 -08:00
let url = reqwest::Url::parse(&format!("{}/api/cameras/{}/main/recordings", server.base_url,
use add_camera in tests, not direct db inserts This is a wash in terms of lines of code now, but it makes it a bit easier to maintain as I make changes to the schema (such as separating out streams from cameras), and it helps ensure the tests reflect reality.
2018-02-03 21:56:04 -08:00
server.test_camera_uuid)).unwrap();
update dependencies
2017-11-16 23:01:09 -08:00
let client = reqwest::Client::new();
fix some reqwest upgrade brokenness in benches
2020-02-16 23:58:07 -08:00
let mut rt = tokio::runtime::Runtime::new().unwrap();
reuse reqwest client in serve_camera_html bench This makes a huge difference in the reported time - 863 usec rather than 6 milliseconds on my laptop. Part of the difference is in reqwest client setup (it apparently initializes a SSL_CTX that is never used in this test), part fresh connections vs keepalive, part I don't know what. None of it seems relevant to the logic I want to test.
2017-03-03 22:26:29 -08:00
let mut f = || {
fix some reqwest upgrade brokenness in benches
2020-02-16 23:58:07 -08:00
rt.block_on(async {
let resp = client.get(url.clone()).send().await.unwrap();
assert_eq!(resp.status(), reqwest::StatusCode::OK);
let _b = resp.bytes().await.unwrap();
});
reuse reqwest client in serve_camera_html bench This makes a huge difference in the reported time - 863 usec rather than 6 milliseconds on my laptop. Part of the difference is in reqwest client setup (it apparently initializes a SSL_CTX that is never used in this test), part fresh connections vs keepalive, part I don't know what. None of it seems relevant to the logic I want to test.
2017-03-03 22:26:29 -08:00
};
f(); // warm.
b.iter(f);
benchmark camera page, fix broken schema This page was noticeably slower than necessary because the recording_cover index wasn't actually covering the query. Both the schema for new databases and the upgrade query were broken (and not even in the same way). No new schema version to correct this, at least for now. I'll probably have another reason to change the schema soon anyway and can throw this in.
2017-02-12 20:37:03 -08:00
}
}
Reference in New Issue Copy Permalink